[ceph-users] Re: Ceph upgrade OSD unsafe to stop

18 Aug 2025

...
'ceph osd ok-to-stop' is a safety check, nothing more.
Oh ok I though that you can add true to ignore pgs becoming inactive
This is the ouput for one OSD , all of my OSD are unsafe to stop.

root@ceph-monitor-1:/# ceph osd ok-to-stop 1
{"ok_to_stop":false,"osds":[1],"num_ok_pgs":30,"num_not_ok_pgs":170,"bad_become_inactive":["2.0","2.2","2.3","2.4","2.7","2.f","2.11","2.14","2.17","2.18","2.19","2.1a","2.1d","3.2","3.5","3.a","3.d","7.0","7.6","7.7","7.8","7.b","7.d","7.11","7.16","7.17","7.19","7.1f","10.2","10.9","10.b","10.c","10.d","10.e","10.10","10.11","10.14","10.1a","10.1b","10.1d","10.1f","11.1","11.4","11.5","11.7","11.9","11.a","11.c","11.d","11.e","11.11","11.13","11.14","11.15","11.16","11.1a","11.1b","11.1e","15.1","15.2","15.9","15.a","15.b","15.d","15.f","15.10","15.11","15.13","15.14","15.16","15.17","15.19","15.1c","15.1d","15.1f","16.2","16.3","16.5","16.6","16.9","16.c","16.d","16.f","16.18","16.19","16.1b","16.1c","16.1d","16.1e","17.6","17.a","17.b","17.c","17.d","17.10","17.11","17.18","17.1f","25.2","25.3","25.5","25.7","25.8","25.b","25.10","25.11","25.13","25.14","25.17","25.19","25.1a","25.1b","25.21","25.22","25.23","25.25","25.27","25.2a","25.2b","25.2e","25.31","25.35","25.37","25.3d","26.1","26.8","26.9","26.a","26.f","26.10","26.14","26.16","26.1a","26.1e","27.0","27.1","27.2","27.4","27.5","27.6","27.c","27.d","27.f","28.2","28.3","28.6","28.7","28.a","28.10","28.14","28.18","28.1a","28.1d","28.1e","28.1f","28.20","28.21","28.25","28.26","28.2a","28.2b","28.2c","28.2d","28.2f","28.33","28.34","28.37","28.3a","28.3d","28.3f"],"ok_become_degraded":["4.0","4.3","4.6","4.8","4.a","4.b","4.d","4.e","4.11","4.13","4.14","4.15","4.17","4.18","4.1b","4.1d","4.1f","18.0","18.2","18.6","18.c","18.d","18.e","18.14","18.16","18.17","18.19","18.1a","18.1d","18.1e"]}
Error EBUSY: unsafe to stop osd(s) at this time (170 PGs are or would become offline)

root@ceph-monitor-1:/# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 886201 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.98
pool 2 'rbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670923 lfor 0/1645/1643 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.19
pool 3 'cephfs.toto-fs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886203 lfor 0/0/64 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.19
pool 4 'cephfs.toto-fs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3008 lfor 0/3008/3006 flags hashpspool,bulk max_bytes 32212254720 stripe_width 0 application cephfs read_balance_score 1.53
pool 7 '.nfs' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886199 lfor 0/0/140 flags hashpspool stripe_width 0 application nfs read_balance_score 1.53
pool 10 'prbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670919 lfor 0/0/833 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53
pool 11 '.rgw.root' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886205 lfor 0/0/833 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53
pool 15 'testzone.rgw.log' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886225 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.97
pool 16 'testzone.rgw.control' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886213 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53
pool 17 'testzone.rgw.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886215 lfor 0/0/1875 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.53
pool 18 'testzone.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1965 lfor 0/0/1877 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.31
pool 25 'pool_VM' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886217 lfor 0/0/885938 flags hashpspool,selfmanaged_snaps max_bytes 107374182400 stripe_width 0 target_size_bytes 536870912000 application rbd read_balance_score 1.64
pool 26 'k8s' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886233 lfor 0/0/886231 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53
pool 27 'cephfs.testfs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886221 lfor 0/0/885939 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.75
pool 28 'cephfs.testfs.data' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886242 lfor 0/0/886240 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.31

root@ceph-monitor-1:/# ceph osd df tree
ID   CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA    OMAP     META     AVAIL    %USE  VAR   PGS  STATUS  TYPE NAME
 -1         6.63879         -  6.4 TiB  104 GiB  95 GiB  164 KiB  9.6 GiB  6.3 TiB  1.60  1.00    -          root inist
-15         0.90970         -  932 GiB   12 GiB  11 GiB   35 KiB  889 MiB  920 GiB  1.29  0.80    -              host ceph-monitor-1
  6    hdd  0.90970   1.00000  932 GiB   12 GiB  11 GiB   35 KiB  889 MiB  920 GiB  1.29  0.80  197      up          osd.6
-21         5.72910         -  5.5 TiB   92 GiB  84 GiB  129 KiB  8.7 GiB  5.4 TiB  1.65  1.03    -              datacenter dcml
-20         2.72910         -  2.7 TiB   46 GiB  42 GiB   62 KiB  4.0 GiB  2.7 TiB  1.64  1.02    -                  room it02
-19         2.72910         -  2.7 TiB   46 GiB  42 GiB   62 KiB  4.0 GiB  2.7 TiB  1.64  1.02    -                      row left
-18         2.72910         -  2.7 TiB   46 GiB  42 GiB   62 KiB  4.0 GiB  2.7 TiB  1.64  1.02    -                          rack 10
 -3         0.90970         -  932 GiB   14 GiB  13 GiB   17 KiB  1.2 GiB  918 GiB  1.50  0.94    -                              host ceph-node-1
  2    hdd  0.90970   1.00000  932 GiB   14 GiB  13 GiB   17 KiB  1.2 GiB  918 GiB  1.50  0.94  201      up                          osd.2
 -5         0.90970         -  932 GiB   16 GiB  14 GiB   28 KiB  1.6 GiB  916 GiB  1.70  1.06    -                              host ceph-node-2
  1    hdd  0.90970   1.00000  932 GiB   16 GiB  14 GiB   28 KiB  1.6 GiB  916 GiB  1.70  1.06  200      up                          osd.1
 -9         0.90970         -  932 GiB   16 GiB  15 GiB   17 KiB  1.2 GiB  915 GiB  1.72  1.07    -                              host ceph-node-3
  5    hdd  0.90970   1.00000  932 GiB   16 GiB  15 GiB   17 KiB  1.2 GiB  915 GiB  1.72  1.07  200      up                          osd.5
-36         3.00000         -  2.7 TiB   47 GiB  42 GiB   67 KiB  4.8 GiB  2.7 TiB  1.67  1.04    -                  room it06
-35         3.00000         -  2.7 TiB   47 GiB  42 GiB   67 KiB  4.8 GiB  2.7 TiB  1.67  1.04    -                      row left06
-34         3.00000         -  2.7 TiB   47 GiB  42 GiB   67 KiB  4.8 GiB  2.7 TiB  1.67  1.04    -                          rack 08
 -7         1.00000         -  932 GiB   15 GiB  13 GiB   21 KiB  1.6 GiB  917 GiB  1.57  0.98    -                              host ceph-node-4
  0    hdd  1.00000   1.00000  932 GiB   15 GiB  13 GiB   21 KiB  1.6 GiB  917 GiB  1.57  0.98  207      up                          osd.0
-13         1.00000         -  932 GiB   15 GiB  14 GiB   31 KiB  1.6 GiB  916 GiB  1.63  1.02    -                              host ceph-node-5
  3    hdd  1.00000   1.00000  932 GiB   15 GiB  14 GiB   31 KiB  1.6 GiB  916 GiB  1.63  1.02  217      up                          osd.3
-11         1.00000         -  932 GiB   17 GiB  15 GiB   15 KiB  1.5 GiB  915 GiB  1.80  1.12    -                              host ceph-node-6
  4    hdd  1.00000   1.00000  932 GiB   17 GiB  15 GiB   15 KiB  1.5 GiB  915 GiB  1.80  1.12  221      up                          osd.4
                        TOTAL  6.4 TiB  104 GiB  95 GiB  168 KiB  9.6 GiB  6.3 TiB  1.60
MIN/MAX VAR: 0.80/1.12  STDDEV: 0.16

Vivien

________________________________
De : Eugen Block <eblock@nde.ag>
Envoyé : lundi 18 août 2025 12:37:14
À : ceph-users@ceph.io
Objet : [ceph-users] Re: Ceph upgrade OSD unsafe to stop

Hi,

'ceph osd ok-to-stop' is a safety check, nothing more. It basically
checks if PGs would become inactive if you stopped said OSD or if
those PGs would become only degraded. Which OSD does report that it's
unsafe to stop? Can you paste the output of 'ceph osd ok-to-stop
<OSD_ID>'? And with that also 'ceph osd pool ls detail' to see which
pool(s) is/are affected. And 'ceph osd df tree' can also be useful here.

Regards,
Eugen

Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
...
Hi,
I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr
upgrade goes well but I had some issue with OSD :
Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would
become offline)
Cluster is in health_ok
All pools are replica 3 and pgs active+clean
autoscaler is off following the ceph docs
Does ceph osd ok-to-stop lead to lost data ?
The only rules used in the cluster is replicated_rule :
root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule
{
    "rule_id": 0,
    "rule_name": "replicated_rule",
    "type": 1,
    "steps": [
        {
            "op": "take",
            "item": -1,
            "item_name": "inist"
        },
        {
            "op": "chooseleaf_firstn",
            "num": 0,
            "type": "host"
        },
        {
            "op": "emit"
        }
    ]
}
root@ceph-monitor-1:/# ceph osd tree
ID   CLASS  WEIGHT   TYPE NAME                             STATUS
REWEIGHT  PRI-AFF
 -1         6.63879  root inist
-15         0.90970      host ceph-monitor-1
  6    hdd  0.90970          osd.6                             up
1.00000  1.00000
-21         5.72910      datacenter bat1
-20         2.72910          room room01
-19         2.72910              row left
-18         2.72910                  rack 10
 -3         0.90970                      host ceph-node-1
  2    hdd  0.90970                          osd.2             up
1.00000  1.00000
 -5         0.90970                      host ceph-node-2
  1    hdd  0.90970                          osd.1             up
1.00000  1.00000
 -9         0.90970                      host ceph-node-3
  5    hdd  0.90970                          osd.5             up
1.00000  1.00000
-36         3.00000          room room03
-35         3.00000              row left06
-34         3.00000                  rack 08
 -7         1.00000                      host ceph-node-4
  0    hdd  1.00000                          osd.0             up
1.00000  1.00000
-13         1.00000                      host ceph-node-5
  3    hdd  1.00000                          osd.3             up
1.00000  1.00000
-11         1.00000                      host ceph-node-6
  4    hdd  1.00000                          osd.4             up
1.00000  1.00000
Thanks !
Vivien
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io