Ceph upgrade OSD unsafe to stop
Hi, I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD : Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline) Cluster is in health_ok All pools are replica 3 and pgs active+clean autoscaler is off following the ceph docs Does ceph osd ok-to-stop lead to lost data ? The only rules used in the cluster is replicated_rule : root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000 Thanks ! Vivien
Hi, 'ceph osd ok-to-stop' is a safety check, nothing more. It basically checks if PGs would become inactive if you stopped said OSD or if those PGs would become only degraded. Which OSD does report that it's unsafe to stop? Can you paste the output of 'ceph osd ok-to-stop <OSD_ID>'? And with that also 'ceph osd pool ls detail' to see which pool(s) is/are affected. And 'ceph osd df tree' can also be useful here. Regards, Eugen Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD :
Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline)
Cluster is in health_ok
All pools are replica 3 and pgs active+clean
autoscaler is off following the ceph docs
Does ceph osd ok-to-stop lead to lost data ?
The only rules used in the cluster is replicated_rule :
root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule
{ "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
'ceph osd ok-to-stop' is a safety check, nothing more.
Oh ok I though that you can add true to ignore pgs becoming inactive This is the ouput for one OSD , all of my OSD are unsafe to stop. root@ceph-monitor-1:/# ceph osd ok-to-stop 1 {"ok_to_stop":false,"osds":[1],"num_ok_pgs":30,"num_not_ok_pgs":170,"bad_become_inactive":["2.0","2.2","2.3","2.4","2.7","2.f","2.11","2.14","2.17","2.18","2.19","2.1a","2.1d","3.2","3.5","3.a","3.d","7.0","7.6","7.7","7.8","7.b","7.d","7.11","7.16","7.17","7.19","7.1f","10.2","10.9","10.b","10.c","10.d","10.e","10.10","10.11","10.14","10.1a","10.1b","10.1d","10.1f","11.1","11.4","11.5","11.7","11.9","11.a","11.c","11.d","11.e","11.11","11.13","11.14","11.15","11.16","11.1a","11.1b","11.1e","15.1","15.2","15.9","15.a","15.b","15.d","15.f","15.10","15.11","15.13","15.14","15.16","15.17","15.19","15.1c","15.1d","15.1f","16.2","16.3","16.5","16.6","16.9","16.c","16.d","16.f","16.18","16.19","16.1b","16.1c","16.1d","16.1e","17.6","17.a","17.b","17.c","17.d","17.10","17.11","17.18","17.1f","25.2","25.3","25.5","25.7","25.8","25.b","25.10","25.11","25.13","25.14","25.17","25.19","25.1a","25.1b","25.21","25.22","25.23","25.25","25.27","25.2a","25.2b","25.2e","25.31","25.35","25.37","25.3d","26.1","26.8","26.9","26.a","26.f","26.10","26.14","26.16","26.1a","26.1e","27.0","27.1","27.2","27.4","27.5","27.6","27.c","27.d","27.f","28.2","28.3","28.6","28.7","28.a","28.10","28.14","28.18","28.1a","28.1d","28.1e","28.1f","28.20","28.21","28.25","28.26","28.2a","28.2b","28.2c","28.2d","28.2f","28.33","28.34","28.37","28.3a","28.3d","28.3f"],"ok_become_degraded":["4.0","4.3","4.6","4.8","4.a","4.b","4.d","4.e","4.11","4.13","4.14","4.15","4.17","4.18","4.1b","4.1d","4.1f","18.0","18.2","18.6","18.c","18.d","18.e","18.14","18.16","18.17","18.19","18.1a","18.1d","18.1e"]} Error EBUSY: unsafe to stop osd(s) at this time (170 PGs are or would become offline) root@ceph-monitor-1:/# ceph osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 886201 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.98 pool 2 'rbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670923 lfor 0/1645/1643 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.19 pool 3 'cephfs.toto-fs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886203 lfor 0/0/64 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.19 pool 4 'cephfs.toto-fs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3008 lfor 0/3008/3006 flags hashpspool,bulk max_bytes 32212254720 stripe_width 0 application cephfs read_balance_score 1.53 pool 7 '.nfs' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886199 lfor 0/0/140 flags hashpspool stripe_width 0 application nfs read_balance_score 1.53 pool 10 'prbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670919 lfor 0/0/833 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53 pool 11 '.rgw.root' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886205 lfor 0/0/833 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53 pool 15 'testzone.rgw.log' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886225 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.97 pool 16 'testzone.rgw.control' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886213 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53 pool 17 'testzone.rgw.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886215 lfor 0/0/1875 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.53 pool 18 'testzone.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1965 lfor 0/0/1877 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.31 pool 25 'pool_VM' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886217 lfor 0/0/885938 flags hashpspool,selfmanaged_snaps max_bytes 107374182400 stripe_width 0 target_size_bytes 536870912000 application rbd read_balance_score 1.64 pool 26 'k8s' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886233 lfor 0/0/886231 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53 pool 27 'cephfs.testfs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886221 lfor 0/0/885939 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.75 pool 28 'cephfs.testfs.data' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886242 lfor 0/0/886240 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.31 root@ceph-monitor-1:/# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 6.63879 - 6.4 TiB 104 GiB 95 GiB 164 KiB 9.6 GiB 6.3 TiB 1.60 1.00 - root inist -15 0.90970 - 932 GiB 12 GiB 11 GiB 35 KiB 889 MiB 920 GiB 1.29 0.80 - host ceph-monitor-1 6 hdd 0.90970 1.00000 932 GiB 12 GiB 11 GiB 35 KiB 889 MiB 920 GiB 1.29 0.80 197 up osd.6 -21 5.72910 - 5.5 TiB 92 GiB 84 GiB 129 KiB 8.7 GiB 5.4 TiB 1.65 1.03 - datacenter dcml -20 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - room it02 -19 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - row left -18 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - rack 10 -3 0.90970 - 932 GiB 14 GiB 13 GiB 17 KiB 1.2 GiB 918 GiB 1.50 0.94 - host ceph-node-1 2 hdd 0.90970 1.00000 932 GiB 14 GiB 13 GiB 17 KiB 1.2 GiB 918 GiB 1.50 0.94 201 up osd.2 -5 0.90970 - 932 GiB 16 GiB 14 GiB 28 KiB 1.6 GiB 916 GiB 1.70 1.06 - host ceph-node-2 1 hdd 0.90970 1.00000 932 GiB 16 GiB 14 GiB 28 KiB 1.6 GiB 916 GiB 1.70 1.06 200 up osd.1 -9 0.90970 - 932 GiB 16 GiB 15 GiB 17 KiB 1.2 GiB 915 GiB 1.72 1.07 - host ceph-node-3 5 hdd 0.90970 1.00000 932 GiB 16 GiB 15 GiB 17 KiB 1.2 GiB 915 GiB 1.72 1.07 200 up osd.5 -36 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - room it06 -35 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - row left06 -34 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - rack 08 -7 1.00000 - 932 GiB 15 GiB 13 GiB 21 KiB 1.6 GiB 917 GiB 1.57 0.98 - host ceph-node-4 0 hdd 1.00000 1.00000 932 GiB 15 GiB 13 GiB 21 KiB 1.6 GiB 917 GiB 1.57 0.98 207 up osd.0 -13 1.00000 - 932 GiB 15 GiB 14 GiB 31 KiB 1.6 GiB 916 GiB 1.63 1.02 - host ceph-node-5 3 hdd 1.00000 1.00000 932 GiB 15 GiB 14 GiB 31 KiB 1.6 GiB 916 GiB 1.63 1.02 217 up osd.3 -11 1.00000 - 932 GiB 17 GiB 15 GiB 15 KiB 1.5 GiB 915 GiB 1.80 1.12 - host ceph-node-6 4 hdd 1.00000 1.00000 932 GiB 17 GiB 15 GiB 15 KiB 1.5 GiB 915 GiB 1.80 1.12 221 up osd.4 TOTAL 6.4 TiB 104 GiB 95 GiB 168 KiB 9.6 GiB 6.3 TiB 1.60 MIN/MAX VAR: 0.80/1.12 STDDEV: 0.16 Vivien ________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : lundi 18 août 2025 12:37:14 À : ceph-users@ceph.io Objet : [ceph-users] Re: Ceph upgrade OSD unsafe to stop Hi, 'ceph osd ok-to-stop' is a safety check, nothing more. It basically checks if PGs would become inactive if you stopped said OSD or if those PGs would become only degraded. Which OSD does report that it's unsafe to stop? Can you paste the output of 'ceph osd ok-to-stop <OSD_ID>'? And with that also 'ceph osd pool ls detail' to see which pool(s) is/are affected. And 'ceph osd df tree' can also be useful here. Regards, Eugen Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD :
Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline)
Cluster is in health_ok
All pools are replica 3 and pgs active+clean
autoscaler is off following the ceph docs
Does ceph osd ok-to-stop lead to lost data ?
The only rules used in the cluster is replicated_rule :
root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule
{ "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, all your pools have size=3 and min_size=3. This is why it is not OK to stop an OSD. Reduce min_size to 2 and you should be able to stop OSDs. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Changed to min_size=2, update is going well now Thanks ! Vivien ________________________________ De : Robert Sander <r.sander@heinlein-support.de> Envoyé : lundi 18 août 2025 13:41:06 À : ceph-users@ceph.io Objet : [ceph-users] Re: Ceph upgrade OSD unsafe to stop Hi, all your pools have size=3 and min_size=3. This is why it is not OK to stop an OSD. Reduce min_size to 2 and you should be able to stop OSDs. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Vivien, what I noticed immediately are your pool settings: You have set on the pools: size 3 *min_size 3* but it should be: size 3 *min_size 2 * The consequence of this is that every time osd fails, the affected pgs go offline. Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 18. Aug. 2025 um 14:13 Uhr schrieb GLE, Vivien <Vivien.GLE@inist.fr
:
'ceph osd ok-to-stop' is a safety check, nothing more.
Oh ok I though that you can add true to ignore pgs becoming inactive This is the ouput for one OSD , all of my OSD are unsafe to stop.
root@ceph-monitor-1:/# ceph osd ok-to-stop 1
{"ok_to_stop":false,"osds":[1],"num_ok_pgs":30,"num_not_ok_pgs":170,"bad_become_inactive":["2.0","2.2","2.3","2.4","2.7","2.f","2.11","2.14","2.17","2.18","2.19","2.1a","2.1d","3.2","3.5","3.a","3.d","7.0","7.6","7.7","7.8","7.b","7.d","7.11","7.16","7.17","7.19","7.1f","10.2","10.9","10.b","10.c","10.d","10.e","10.10","10.11","10.14","10.1a","10.1b","10.1d","10.1f","11.1","11.4","11.5","11.7","11.9","11.a","11.c","11.d","11.e","11.11","11.13","11.14","11.15","11.16","11.1a","11.1b","11.1e","15.1","15.2","15.9","15.a","15.b","15.d","15.f","15.10","15.11","15.13","15.14","15.16","15.17","15.19","15.1c","15.1d","15.1f","16.2","16.3","16.5","16.6","16.9","16.c","16.d","16.f","16.18","16.19","16.1b","16.1c","16.1d","16.1e","17.6","17.a","17.b","17.c","17.d","17.10","17.11","17.18","17.1f","25.2","25.3","25.5","25.7","25.8","25.b","25.10","25.11","25.13","25.14","25.17","25.19","25.1a","25.1b","25.21","25.22","25.23","25.25","25.27","25.2a","25.2b","25.2e","25.31","25.35","25.37","25.3d","26.1","26.8","26.9","26.a","26.f","26.10","26.14","26.16","26.1a","26.1e","27.0","27.1","27.2","27.4","27.5","27.6","27.c","27.d","27.f","28.2","28.3","28.6","28.7","28.a","28.10","28.14","28.18","28.1a","28.1d","28.1e","28.1f","28.20","28.21","28.25","28.26","28.2a","28.2b","28.2c","28.2d","28.2f","28.33","28.34","28.37","28.3a","28.3d","28.3f"],"ok_become_degraded":["4.0","4.3","4.6","4.8","4.a","4.b","4.d","4.e","4.11","4.13","4.14","4.15","4.17","4.18","4.1b","4.1d","4.1f","18.0","18.2","18.6","18.c","18.d","18.e","18.14","18.16","18.17","18.19","18.1a","18.1d","18.1e"]} Error EBUSY: unsafe to stop osd(s) at this time (170 PGs are or would become offline)
root@ceph-monitor-1:/# ceph osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 886201 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.98 pool 2 'rbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670923 lfor 0/1645/1643 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.19 pool 3 'cephfs.toto-fs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886203 lfor 0/0/64 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.19 pool 4 'cephfs.toto-fs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3008 lfor 0/3008/3006 flags hashpspool,bulk max_bytes 32212254720 stripe_width 0 application cephfs read_balance_score 1.53 pool 7 '.nfs' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886199 lfor 0/0/140 flags hashpspool stripe_width 0 application nfs read_balance_score 1.53 pool 10 'prbd' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 670919 lfor 0/0/833 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53 pool 11 '.rgw.root' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886205 lfor 0/0/833 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53 pool 15 'testzone.rgw.log' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886225 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.97 pool 16 'testzone.rgw.control' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886213 lfor 0/0/1153 flags hashpspool stripe_width 0 application rgw read_balance_score 1.53 pool 17 'testzone.rgw.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886215 lfor 0/0/1875 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.53 pool 18 'testzone.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1965 lfor 0/0/1877 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.31 pool 25 'pool_VM' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886217 lfor 0/0/885938 flags hashpspool,selfmanaged_snaps max_bytes 107374182400 stripe_width 0 target_size_bytes 536870912000 application rbd read_balance_score 1.64 pool 26 'k8s' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 886233 lfor 0/0/886231 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.53 pool 27 'cephfs.testfs.meta' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 886221 lfor 0/0/885939 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.75 pool 28 'cephfs.testfs.data' replicated size 3 min_size 3 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 886242 lfor 0/0/886240 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.31
root@ceph-monitor-1:/# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 6.63879 - 6.4 TiB 104 GiB 95 GiB 164 KiB 9.6 GiB 6.3 TiB 1.60 1.00 - root inist -15 0.90970 - 932 GiB 12 GiB 11 GiB 35 KiB 889 MiB 920 GiB 1.29 0.80 - host ceph-monitor-1 6 hdd 0.90970 1.00000 932 GiB 12 GiB 11 GiB 35 KiB 889 MiB 920 GiB 1.29 0.80 197 up osd.6 -21 5.72910 - 5.5 TiB 92 GiB 84 GiB 129 KiB 8.7 GiB 5.4 TiB 1.65 1.03 - datacenter dcml -20 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - room it02 -19 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - row left -18 2.72910 - 2.7 TiB 46 GiB 42 GiB 62 KiB 4.0 GiB 2.7 TiB 1.64 1.02 - rack 10 -3 0.90970 - 932 GiB 14 GiB 13 GiB 17 KiB 1.2 GiB 918 GiB 1.50 0.94 - host ceph-node-1 2 hdd 0.90970 1.00000 932 GiB 14 GiB 13 GiB 17 KiB 1.2 GiB 918 GiB 1.50 0.94 201 up osd.2 -5 0.90970 - 932 GiB 16 GiB 14 GiB 28 KiB 1.6 GiB 916 GiB 1.70 1.06 - host ceph-node-2 1 hdd 0.90970 1.00000 932 GiB 16 GiB 14 GiB 28 KiB 1.6 GiB 916 GiB 1.70 1.06 200 up osd.1 -9 0.90970 - 932 GiB 16 GiB 15 GiB 17 KiB 1.2 GiB 915 GiB 1.72 1.07 - host ceph-node-3 5 hdd 0.90970 1.00000 932 GiB 16 GiB 15 GiB 17 KiB 1.2 GiB 915 GiB 1.72 1.07 200 up osd.5 -36 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - room it06 -35 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - row left06 -34 3.00000 - 2.7 TiB 47 GiB 42 GiB 67 KiB 4.8 GiB 2.7 TiB 1.67 1.04 - rack 08 -7 1.00000 - 932 GiB 15 GiB 13 GiB 21 KiB 1.6 GiB 917 GiB 1.57 0.98 - host ceph-node-4 0 hdd 1.00000 1.00000 932 GiB 15 GiB 13 GiB 21 KiB 1.6 GiB 917 GiB 1.57 0.98 207 up osd.0 -13 1.00000 - 932 GiB 15 GiB 14 GiB 31 KiB 1.6 GiB 916 GiB 1.63 1.02 - host ceph-node-5 3 hdd 1.00000 1.00000 932 GiB 15 GiB 14 GiB 31 KiB 1.6 GiB 916 GiB 1.63 1.02 217 up osd.3 -11 1.00000 - 932 GiB 17 GiB 15 GiB 15 KiB 1.5 GiB 915 GiB 1.80 1.12 - host ceph-node-6 4 hdd 1.00000 1.00000 932 GiB 17 GiB 15 GiB 15 KiB 1.5 GiB 915 GiB 1.80 1.12 221 up osd.4 TOTAL 6.4 TiB 104 GiB 95 GiB 168 KiB 9.6 GiB 6.3 TiB 1.60 MIN/MAX VAR: 0.80/1.12 STDDEV: 0.16
Vivien
________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : lundi 18 août 2025 12:37:14 À : ceph-users@ceph.io Objet : [ceph-users] Re: Ceph upgrade OSD unsafe to stop
Hi,
'ceph osd ok-to-stop' is a safety check, nothing more. It basically checks if PGs would become inactive if you stopped said OSD or if those PGs would become only degraded. Which OSD does report that it's unsafe to stop? Can you paste the output of 'ceph osd ok-to-stop <OSD_ID>'? And with that also 'ceph osd pool ls detail' to see which pool(s) is/are affected. And 'ceph osd df tree' can also be useful here.
Regards, Eugen
Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm trying to update my cluster (19.2.2 -> 19.2.3), mon and mgr upgrade goes well but I had some issue with OSD :
Upgrade: unsafe to stop osd(s) at this time (165 PGs are or would become offline)
Cluster is in health_ok
All pools are replica 3 and pgs active+clean
autoscaler is off following the ceph docs
Does ceph osd ok-to-stop lead to lost data ?
The only rules used in the cluster is replicated_rule :
root@ceph-monitor-1:/# ceph osd crush rule dump replicated_rule
{ "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "inist" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
root@ceph-monitor-1:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 6.63879 root inist -15 0.90970 host ceph-monitor-1 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -21 5.72910 datacenter bat1 -20 2.72910 room room01 -19 2.72910 row left -18 2.72910 rack 10 -3 0.90970 host ceph-node-1 2 hdd 0.90970 osd.2 up 1.00000 1.00000 -5 0.90970 host ceph-node-2 1 hdd 0.90970 osd.1 up 1.00000 1.00000 -9 0.90970 host ceph-node-3 5 hdd 0.90970 osd.5 up 1.00000 1.00000 -36 3.00000 room room03 -35 3.00000 row left06 -34 3.00000 rack 08 -7 1.00000 host ceph-node-4 0 hdd 1.00000 osd.0 up 1.00000 1.00000 -13 1.00000 host ceph-node-5 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph-node-6 4 hdd 1.00000 osd.4 up 1.00000 1.00000
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
GLE, Vivien
-
Joachim Kraftmayer
-
Robert Sander