Den tors 4 apr. 2024 kl 06:11 skrev Zakhar Kirpichenko <zakhar@gmail.com>:

Any comments regarding `osd noin`, please?

...

I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`:

cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set

Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`:

Only that we mostly set "initial osd crush weight" to 0.0001 so they are up and in at first start, but don't receive PGs, instead of using "noin" when adding new OSDs. -- May the most significant bit of your life be positive.

Hi, the noin flag seems to be only applicable to existing OSDs which are already in the crushmap. It doesn't apply to newly created OSDs, I could confirm that in a small test cluster with Pacific and Reef. I don't have any insights if that is by design or not, I assume it's supposed to work like that. If you want to prevent data movement when creating new OSDs you could use the osd_crush_initial_weight config option and set it to 0. We have that in our cluster as well, but of course you'd have to reweight new OSDs manually. Regards, Eugen Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:

Any comments regarding `osd noin`, please?

/Z

On Tue, 2 Apr 2024 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:

...

Hi,

I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`:

cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set

Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`:

services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin

# ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000

Is this expected behavior? Do I misunderstand the purpose of the `noin` option?

Best regards, Zakhar

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, Another way to prevent data movement at OSD creation time (appart from using norebalance and nobackfill cluster flags) is to pre-create the host buckets in another root, named for example "closet", let the orchestrator create the OSDs and move these host buckets to their final bucket location in the main root, when you're ready to (when all new OSDs are up and in).   This will help when adding new hosts to the cluster, but when adding new OSDs to existing hosts, you'll still have to play with weights and flags. Regards, Frédéric. ________________________________ De : Zakhar Kirpichenko <zakhar@gmail.com> Envoyé : jeudi 4 avril 2024 10:18 À : Eugen Block Cc: ceph-users@ceph.io Objet : [ceph-users] Re: Pacific 16.2.15 `osd noin` Thank you, Eugen. This makes sense. /Z On Thu, 4 Apr 2024 at 10:32, Eugen Block <eblock@nde.ag> wrote:

Hi,

the noin flag seems to be only applicable to existing OSDs which are already in the crushmap. It doesn't apply to newly created OSDs, I could confirm that in a small test cluster with Pacific and Reef. I don't have any insights if that is by design or not, I assume it's supposed to work like that. If you want to prevent data movement when creating new OSDs you could use the osd_crush_initial_weight config option and set it to 0. We have that in our cluster as well, but of course you'd have to reweight new OSDs manually.

Regards, Eugen

Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:

...

Any comments regarding `osd noin`, please?

/Z

On Tue, 2 Apr 2024 at 16:09, Zakhar Kirpichenko <zakhar@gmail.com> wrote:

...

Hi,

I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`:

cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set

Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`:

services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin

# ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000

Is this expected behavior? Do I misunderstand the purpose of the `noin` option?

Best regards, Zakhar

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io