- ceph-users - lists.ceph.io

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Anthony D'Atri 24 Jun '26

24 Jun '26

Honestly, this sort of thing is what led me a while back to eschew the replication network whenever possible. I have a cluster today from which I'm going to remove it. > On Jun 24, 2026, at 5:50 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs? > > It seems that it's not enabled by default. > > And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting. I *suspect* that that wouldn't have prevented this particular situation, but it does inform the idea of incremental deployment of OSDs. Back in the day I tripled the size of a Dumpling cluster. It was not pretty.

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Eugen Block 24 Jun '26

24 Jun '26

Hi, since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs? It seems that it's not enabled by default. And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting. Thanks, Eugen Zitat von Alex Gorbachev via ceph-users <ceph-users(a)ceph.io>: > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made independently of monitor quorum health. An OSD > can begin reporting peers as down based on heartbeat failures even when > monitor quorum remains healthy. > I was adding a second rack of OSD servers to an existing installation, > assuming that the worst that could happen is that those will be > unreachable. I also missed setting up the cluster_network VLAN on one side > (in the 4x100GbE bond). > > I added all the new OSDs at once. The moment that happened, the existing > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > The take away in my recommendation is to add OSDs in such a way that > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to > fix connectivity issues before dealing with down OSDs, as I experienced. > -- > Alex Gorbachev > ISS/Storcium > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph cephadm config-check status
by Eugen Block 24 Jun '26

24 Jun '26

Hi, I noticed that the 'ceph cephadm config-check ls' output doesn't match the 'ceph cephadm config-check status' output, at least it's not what I would have expected. For example, the "STATUS" of all the checks is enabled according to the ls output: ceph cephadm config-check ls NAME HEALTHCHECK STATUS DESCRIPTION kernel_security CEPHADM_CHECK_KERNEL_LSM enabled checks SELINUX/Apparmor profiles are consistent across cluster hosts os_subscription CEPHADM_CHECK_SUBSCRIPTION enabled checks subscription states are consistent for all cluster hosts public_network CEPHADM_CHECK_PUBLIC_MEMBERSHIP enabled check that all hosts have a NIC on the Ceph public_network osd_mtu_size CEPHADM_CHECK_MTU enabled check that OSD hosts share a common MTU setting osd_linkspeed CEPHADM_CHECK_LINKSPEED enabled check that OSD hosts share a common linkspeed network_missing CEPHADM_CHECK_NETWORK_MISSING enabled checks that the cluster/public networks defined exist on the Ceph hosts ceph_release CEPHADM_CHECK_CEPH_RELEASE enabled check for Ceph version consistency - ceph daemons should be on the same release (unless upgrade is active) kernel_version CEPHADM_CHECK_KERNEL_VERSION enabled checks that the MAJ.MIN of the kernel on Ceph hosts is consistent But 'config-check status' shows disabled: ceph cephadm config-check status Disabled I assume that the config-checks would only become active after enabling it: ceph config set mgr mgr/cephadm/config_checks_enabled true ceph cephadm config-check status Enabled Is that assumption correct? I would expect the ls output not to show the STATUS "enabled" unless "config_checks_enabled" is actually enabled. Thanks, Eugen

1 0

Re: Advice needed: CephFS PGs down/unknown after several OSDs failure — implications on marking OSDs lost
by Eugen Block 24 Jun '26

24 Jun '26

Hi, I don't really have simple answers here, but I'll comment inline. Zitat von Alessandro De Salvo via ceph-users <ceph-users(a)ceph.io>: > Dear ceph-users, > > we are looking for an advice before taking any destructive action on > a production Ceph cluster (18.2.4) after a SAN failure. > > We lost access to a Disk System hosting several OSDs. The bulk of > the recovery/backfills did complete, and most PGs are now > `active+clean`, but a small set of PGs remains in status > `unavailable`. The Disk System might still be recoverable, but not > easily or quickly; it may take days, or it may never come back. We > are trying to understand the risk of waiting versus marking the > missing OSDs as lost. > > We have not marked any OSDs as lost yet. > > This is a Ceph Reef cluster, version 18.2.4. > > Sanitized current status: > > health: HEALTH_WARN > 1 filesystem is degraded > 1 MDS reports slow metadata IOs > noscrub,nodeep-scrub flags set > (temporarily enabled by us to reduce scrub load during recovery/rebalancing) > Reduced data availability: 28 PGs inactive, 12 PGs down > slow ops still present > osd: > 86 OSDs total > 69 up/in > 17 OSDs unavailable because of the SAN issue > pgs: > 3320 active+clean > 16 unknown > 11 down > 1 down+remapped > > The problematic PGs are spread across several services/pools: > > Main CephFS data pool: 4 PGs down > Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown > RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown > RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown > RBD/Kubernetes: 1 PG unknown Could you share the following output: ceph osd tree ceph osd pool ls detail ceph health detail > The services we care about most are the filesystems, especially the > main CephFS and the secondary CephFS. > > For the main CephFS, the metadata pool does not appear to have > inactive PGs. The remaining problematic PGs are in the data pool. Of > the main CephFS data PGs, only one appears to contain real data; the > others appear empty. I find it very unlikely that only one of the PGs has data. If there are down OSDs, they can't report to the mgr, so the stats are most likely wrong. > For the non-empty main CephFS data PG, `pg query` shows roughly: > > state: down > acting/up: all current acting OSDs are up > last_update == last_complete > empty: 0 > incomplete: 0 > stat_sum: > num_bytes: ~80 GB > num_objects: ~131k > num_objects_missing: 0 > num_objects_unfound: 0 > num_objects_degraded: 0 > num_objects_misplaced: 0 > blocked: > peering is blocked due to down OSDs > down_osds_we_would_probe: > a few OSDs from previous intervals > peering_blocked_by: > "starting or marking this osd lost may let us proceed" > > The recovery state for this PG includes past intervals with OSDs > that are currently unavailable. The current acting set is up, but > Ceph refuses to activate the PG because it still wants to probe down > OSDs from the PG history. > > We tried to list objects from that PG: > rados -p <main_cephfs_data_pool> ls --pgid <pgid> > > but it hangs/times out, so we cannot currently map RADOS objects > back to CephFS inode/path names. > > For the secondary CephFS, the situation is more concerning because > one metadata PG is `unknown`. Most `pg query` attempts on unknown > PGs return: > Error ENOENT: i don't have pgid <pgid> Same applies here, down OSDs can't report anything. > For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and > down, while some others appear empty. Some metadata PGs are unknown. > RGW can temporarily work after restarting daemons, but eventually > gets stuck again, presumably when requests hit unavailable RADOS > objects or metadata. > > We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD > pool, only one non-critical PVC seems to hang on `rbd info`; most > other images respond. So the main operational concern remains CephFS > and RGW. > > Our questions are: > 1. In a case like this, where a CephFS data PG is `down`, not > `incomplete`, and shows `num_objects_missing = 0` and > `num_objects_unfound = 0`, but peering is blocked by historical down > OSDs, what are the likely consequences of marking those OSDs as lost? Data loss. IIRC, a down OSD means there are no OSDs to query at all (for that PG) while incomplete means that there are not enough OSDs "up" to determine which one has the most up-to-date state. > 2. If we mark the unavailable OSDs as lost, should we expect the > main CephFS to: > - come back with only some files or file extents unreadable/corrupt, > - remain blocked/degraded, > - or risk a wider filesystem-level failure? That's difficult to answer. Losing an entire OSD can mean several TB of data lost, so it's unlikely that "only some files or file extents" will be lost. I think it would allow Ceph to recover further, most likely you'd have to scrub the FS afterwards, but at some point you should have a working FS, I think. > 3. Since the main CephFS metadata pool does not appear to have > inactive PGs, is it reasonable to assume that the risk is mostly > limited to file data objects rather than the entire namespace? Or is > that assumption unsafe? I tend to concur, but what do you mean by "does not appear to have inactive PGs"? The 'ceph health detail' output should have a clear answer to that. > 4. For the secondary CephFS, where a metadata PG is `unknown`, > should we treat declaring OSDs lost as much more dangerous? Could > that make the whole filesystem unusable rather than only losing some > files? There are ways to recover the metadata, but be very careful with that! https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ > 5. Is there any recommended way to map the affected CephFS data PG > to file paths while the PG is still down? `rados ls --pgid` hangs, > so we currently cannot list objects in the PG. I don't think so. > 6. Before declaring the unavailable OSDs lost, are there any > additional diagnostic or recovery steps you would recommend to > better assess the situation or reduce the risk of data loss? Is there no chance to access the missing OSDs? So one could at least export PGs or something? > Any advice would be greatly appreciated. > > Best regards, > > Nadir and Alessandro > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSD don't ant to start
by Eugen Block 24 Jun '26

24 Jun '26

Although I wouldn't have expected it to take 20 restarts, it's still good that it helped resolving the issue. I'm not entirely sure of the implications having increased osd_map_share_max_epochs that much. Did you set it back to the default? If not, I recommend to monitor disk usage closely (monitor store), and maybe OSD memory usage as well. Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via > ceph-users a écrit : >> Hi, >> >> this sounds familiar: >> >> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA7 >> GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO >> >> You could set the OSD "in" and then retry starting it a couple of times. >> >> Regards, >> Eugen > > Yes that sound really that. > There's a RedHat solution about that (behind a closed wall), they suggest to > set to 1000 thoses settings : > osd_map_share_max_epochs and another (I can't access it from here...). > > It didn't work at first so I thought it wasn't it. > But I certainly was far behind more than 1000... > My OSD stayed down/out for at least 4 days. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSD don't ant to start
by Gilles Mocellin 23 Jun '26

23 Jun '26

Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via ceph-users a écrit : > Hi, > > this sounds familiar: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA7 > GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO > > You could set the OSD "in" and then retry starting it a couple of times. > > Regards, > Eugen Yes that sound really that. There's a RedHat solution about that (behind a closed wall), they suggest to set to 1000 thoses settings : osd_map_share_max_epochs and another (I can't access it from here...). It didn't work at first so I thought it wasn't it. But I certainly was far behind more than 1000... My OSD stayed down/out for at least 4 days.

1 0

Re: OSD don't ant to start
by Gilles Mocellin 23 Jun '26

23 Jun '26

Le mardi 23 juin 2026, 10:29:07 heure d’été d’Europe centrale Gilles Mocellin via ceph-users a écrit : > Hello, > > After removing a failing drive with : > ceph orch osd rm ${ID} --zap --replace > > Waiting for recovery (3 days, in a 12 nodes 144 drives)... > > I add the drive with : > ceph orch daemon add osd myserver:/dev/sdn > > In the journal logs of the OSD, I can see : > [...] OSD shutdown > I've found that solution on RedHat site : > https://access.redhat.com/solutions/7029908 > > Which is related to epochs, they have that log before : > 276003 handle_osd_map epochs [276004,276043] i have 276003, src has > [274883, 303268] > 276043 map says i am stopped by admin. shutting down. > > I don't have messages about epochs. > > I tried anyway with thoses settings : > # ceph config set osd osd_map_message_max 1000 > # ceph config set osd osd_map_share_max_epochs 1000 > > But no, the OSD still doesn't start. I try to restart several times again, and see that an ID (seems to be the osdmap epoch) is increasing each time. After something like 20 restarts (systemd doesn't like that, I had to reset- failed the service several times), I reached the last_commited osdmap and the OSD starts !

1 0

PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 23 Jun '26

23 Jun '26

osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond). I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down. The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium

1 0

Re: Ceph Scrub Schedule
by Alex 23 Jun '26

23 Jun '26

Hi Eugen, we do almost the same thing. Funny that we came up with the same approach.

1 0

Re: OSD don't ant to start
by Eugen Block 23 Jun '26

23 Jun '26

Hi, this sounds familiar: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA… You could set the OSD "in" and then retry starting it a couple of times. Regards, Eugen Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > Hello, > > After removing a failing drive with : > ceph orch osd rm ${ID} --zap --replace > > Waiting for recovery (3 days, in a 12 nodes 144 drives)... > > I add the drive with : > ceph orch daemon add osd myserver:/dev/sdn > > In the journal logs of the OSD, I can see : > > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 start_boot > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_max_backfills set to 1 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active set to 0 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active_hdd set to 3 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active_ssd set to 10 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 bench > count 12288000 bsize 4 KiB > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth > (MiB/sec): 6.518 iops: 1668.622 elapsed_sec: 1.798 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1fffa90740 0 log_channel(cluster) > log [WRN] : OSD bench result of 1668.621657 IOPS is not within the > threshold limit range of 50.000000 IOPS and 500.000000 IOPS for > osd.66. IOPS capacity is unchanged at 315.000000 IOPS. The > recommendation is to establish the osd's IOPS capacity using other > benchmark tools (e.g. Fio) and then override > osd_mclock_max_capacity_iops_[hdd|ssd]. > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1ffba10640 -1 osd.66 389707 osdmap > says I am destroyed > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1fef00a640 -1 osd.66 389747 map says > i am stopped by admin. shutting down. > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1fef00a640 0 osd.66 389747 > _committed_osd_maps shutdown OSD via async signal > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 received signal: > Interrupt from Kernel ( Could be generated by pthread_kill(), > raise(), abort(), alarm() ) UID: 0 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** Got > signal Interrupt *** > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 Fast > Shutdown: - cct->_conf->osd_fast_shutdown = 1, null-fm = 0 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** > Immediate shutdown (osd_fast_shutdown=true) *** > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 > prepare_to_stop starting shutdown > Jun 23 10:06:57 myserver systemd[1]: > ceph-02acb271-d31a-4c26-82a2-0e0e1c579ed5(a)osd.66.service: > Deactivated successfully. > > I retry several time, nothing new. > The OSD is mark down and then stops. > > I've found that solution on RedHat site : > https://access.redhat.com/solutions/7029908 > > Which is related to epochs, they have that log before : > 276003 handle_osd_map epochs [276004,276043] i have 276003, src has > [274883, 303268] > 276043 map says i am stopped by admin. shutting down. > > I don't have messages about epochs. > > I tried anyway with thoses settings : > # ceph config set osd osd_map_message_max 1000 > # ceph config set osd osd_map_share_max_epochs 1000 > > But no, the OSD still doesn't start. > > Has anyone a clue ? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0