I have been slowly migrating towards spec files as I prefer declarative management as a rule. However, I think that we may have a dichotomy in the user base. On the one hand, users with dozens/hundreds of server/drives of basically identical character. On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage. Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where. At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.    Tim On 10/31/24 14:28, Eugen Block wrote:

Hi,

the preferred method to deploy OSDs in cephadm managed clusters are spec files, see this part of the docs [0] for more information. I would just not use the '--all-available-devices' flag, except in test clusters, or if you're really sure that this is what you want.

If you use 'ceph orch daemon add osd ...', you'll end up with one (or more) OSD(s), but they will be unmanaged, as you already noted in your own cluster. There are a couple of examples with advanced specs (e. g. DB/WAL on dedicated devices) in the docs as well [1]. So my recommendation would be to have a suiting spec file for your disk layout. You can always check with the '--dry-run' flag before actually applying it:

ceph orch apply -i osd-spec.yaml --dry-run

Regards, Eugen

[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds [1] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

Zitat von Tim Holloway <timh@mousetech.com>:

...

As I understand it, the manual OSD setup is only for legacy (non-container) OSDs. Directory locations are wrong for managed (containerized) OSDs, for one.

Actually, the whole manual setup docs ought to be moved out of the mainline documentation. In their present arrangement, they make legacy setup sound like the preferred method. And have you noticed that there is no corresponding well-marked section titled "Authomated (cephadmin) setup?".

This is how we end up with OSDs that are simultaneously legacy AND administered for the same OSD, since at last count there are no interlocks within Ceph to prevent such a mess.

   Tim

On 10/31/24 13:39, Dave Hall wrote:

...

Hello.

Sorry if it appears that I am reposting the same issue under a different topic.  However, I feel that the problem has moved and I now have different questions.

At this point I have, I believe, removed all traces of OSD.12 from my cluster - based on steps in the Reef docs at https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#.  I have further located and removed the WAL/DB LV on an associated NVMe drive (shared with 3 other OSDs).

I don't believe the instructions for replacing an OSD (ceph-volume lvm prepare) still apply, so I have been trying to work with the instructions under ADDING AN OSD (MANUAL).

However, since my installation is containerized (Podman), it is unclear which steps should be issued on the host and which within 'cephadm shell'.

There is also another ambiguity:  In step 3 the instruction is to 'mkfs -t {fstype}' and then to 'mount -o user_xattr'.  However, which fs type?

After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets throws errors about the keyring file.

So, are these the right instructions to be using in a containerized installation?  Are there, in general, alternate documents for containerized installations?

Lastly, the above cited instructions don't say anything about the separate WAL/DB LV.

Please advise.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy. Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me. But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take. Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes, I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.    Tim On 10/31/24 15:39, Eugen Block wrote:

I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.

It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.

Zitat von Tim Holloway <timh@mousetech.com>:

...

I have been slowly migrating towards spec files as I prefer declarative management as a rule.

However, I think that we may have a dichotomy in the user base.

On the one hand, users with dozens/hundreds of server/drives of basically identical character.

On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage.

Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where.

At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.

   Tim

On 10/31/24 14:28, Eugen Block wrote:

...

Hi,

the preferred method to deploy OSDs in cephadm managed clusters are spec files, see this part of the docs [0] for more information. I would just not use the '--all-available-devices' flag, except in test clusters, or if you're really sure that this is what you want.

If you use 'ceph orch daemon add osd ...', you'll end up with one (or more) OSD(s), but they will be unmanaged, as you already noted in your own cluster. There are a couple of examples with advanced specs (e. g. DB/WAL on dedicated devices) in the docs as well [1]. So my recommendation would be to have a suiting spec file for your disk layout. You can always check with the '--dry-run' flag before actually applying it:

ceph orch apply -i osd-spec.yaml --dry-run

Regards, Eugen

[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds [1] https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

Zitat von Tim Holloway <timh@mousetech.com>:

...

As I understand it, the manual OSD setup is only for legacy (non-container) OSDs. Directory locations are wrong for managed (containerized) OSDs, for one.

Actually, the whole manual setup docs ought to be moved out of the mainline documentation. In their present arrangement, they make legacy setup sound like the preferred method. And have you noticed that there is no corresponding well-marked section titled "Authomated (cephadmin) setup?".

This is how we end up with OSDs that are simultaneously legacy AND administered for the same OSD, since at last count there are no interlocks within Ceph to prevent such a mess.

   Tim

On 10/31/24 13:39, Dave Hall wrote:

...

Hello.

Sorry if it appears that I am reposting the same issue under a different topic.  However, I feel that the problem has moved and I now have different questions.

At this point I have, I believe, removed all traces of OSD.12 from my cluster - based on steps in the Reef docs at https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#.  I have further located and removed the WAL/DB LV on an associated NVMe drive (shared with 3 other OSDs).

I don't believe the instructions for replacing an OSD (ceph-volume lvm prepare) still apply, so I have been trying to work with the instructions under ADDING AN OSD (MANUAL).

However, since my installation is containerized (Podman), it is unclear which steps should be issued on the host and which within 'cephadm shell'.

There is also another ambiguity:  In step 3 the instruction is to 'mkfs -t {fstype}' and then to 'mount -o user_xattr'.  However, which fs type?

After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets throws errors about the keyring file.

So, are these the right instructions to be using in a containerized installation?  Are there, in general, alternate documents for containerized installations?

Lastly, the above cited instructions don't say anything about the separate WAL/DB LV.

Please advise.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I can't offer a spec off the cuff, but if the LV still exists and you don't need to change its size, then I'd zap it to remove residual Ceph info because otherwise the operation will complain and fail. Having done that, the requirements should be the same as a first-time construction of an OSD on that LV. Eugen can likely give you the spec info. I'd have to RTFM.    Tim On 11/1/24 11:22, Dave Hall wrote:

Tim, Eugen,

So what would a spec file look like for a single OSD that uses a specific HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe drive? Regarding the NVMe, there are 3 other OSDs already using 25% each of this NVMe for WAL/DB, but I have removed the LV that was used by the failed OSD. Do I need to pre-create the LV, or will 'ceph orch' do that for me?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@mousetech.com> wrote:

...

I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy.

Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me.

But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take.

Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes,

I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.

Tim

On 10/31/24 15:39, Eugen Block wrote:

...

I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.

It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.

Zitat von Tim Holloway <timh@mousetech.com>:

...

I have been slowly migrating towards spec files as I prefer declarative management as a rule.

However, I think that we may have a dichotomy in the user base.

On the one hand, users with dozens/hundreds of server/drives of basically identical character.

On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage.

Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where.

At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.

Tim

On 10/31/24 14:28, Eugen Block wrote:

...

Hi,

the preferred method to deploy OSDs in cephadm managed clusters are spec files, see this part of the docs [0] for more information. I would just not use the '--all-available-devices' flag, except in test clusters, or if you're really sure that this is what you want.

If you use 'ceph orch daemon add osd ...', you'll end up with one (or more) OSD(s), but they will be unmanaged, as you already noted in your own cluster. There are a couple of examples with advanced specs (e. g. DB/WAL on dedicated devices) in the docs as well [1]. So my recommendation would be to have a suiting spec file for your disk layout. You can always check with the '--dry-run' flag before actually applying it:

ceph orch apply -i osd-spec.yaml --dry-run

Regards, Eugen

[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds [1]

https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

...

Zitat von Tim Holloway <timh@mousetech.com>:

...

As I understand it, the manual OSD setup is only for legacy (non-container) OSDs. Directory locations are wrong for managed (containerized) OSDs, for one.

Actually, the whole manual setup docs ought to be moved out of the mainline documentation. In their present arrangement, they make legacy setup sound like the preferred method. And have you noticed that there is no corresponding well-marked section titled "Authomated (cephadmin) setup?".

This is how we end up with OSDs that are simultaneously legacy AND administered for the same OSD, since at last count there are no interlocks within Ceph to prevent such a mess.

Tim

On 10/31/24 13:39, Dave Hall wrote: > Hello. > > Sorry if it appears that I am reposting the same issue under a > different > topic. However, I feel that the problem has moved and I now have > different > questions. > > At this point I have, I believe, removed all traces of OSD.12 from my > cluster - based on steps in the Reef docs at > https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. > I have > further located and removed the WAL/DB LV on an associated NVMe drive > (shared with 3 other OSDs). > > I don't believe the instructions for replacing an OSD (ceph-volume > lvm > prepare) still apply, so I have been trying to work with the > instructions > under ADDING AN OSD (MANUAL). > > However, since my installation is containerized (Podman), it is > unclear > which steps should be issued on the host and which within 'cephadm > shell'. > > There is also another ambiguity: In step 3 the instruction is to > 'mkfs -t > {fstype}' and then to 'mount -o user_xattr'. However, which fs type? > > After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets > throws errors about the keyring file. > > So, are these the right instructions to be using in a containerized > installation? Are there, in general, alternate documents for > containerized > installations? > > Lastly, the above cited instructions don't say anything about the > separate > WAL/DB LV. > > Please advise. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall@binghamton.edu > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I'm afraid I've never paid much attention to "ceprh orch ls osd". Mostly I look at ceph orch ps or ceph osd tree and I nenver noticed whether they had a specific indicator for managed/unmanaged. The spec file is bog-standard YAML (Yet Another Markup Language) and the actual valid elements for a given spec are defined by its consumer. There's very good documentation on the spec info for OSDs. I think that Ceph can actually run off a master spec file with all of the options for all types of resources in it, but I cannot be certain. "*" is simply a wildcard pattern meaning "all servers". If I had, for example only certain hosts reserved for OSDs, say 'osd01.ceph.mousetech.com', then a pattern like 'osd*.ceph.mousetech.com' might work. OSDs do not automatically convert. You have to issue an explicit cephadm command to make a legacy OSD become an administered one. And it will fail if you've done something like double-define an OSD. Tim On Fri, 2024-11-01 at 14:28 -0400, Dave Hall wrote:

Tim,

Actually, the links the Eugen shared earlier were sufficient.  I ended up with

service_type: osd service_name: osd placement:   host_pattern: 'ceph01' spec:   data_devices:     rotational: 1   db_devices:     rotational: 0

This worked exactly right as far as creating the OSD - it found and reused the same OSD number that was previously destroyed, and also recreated the WAL/DB LV using the 'blank spot' on the NVMe drive.

However, I'm a bit concerned that the output of 'ceph orch ls osd' has changed in a way that might not be quite good:

NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT osd               32  3m ago     52m  ceph01

Before all of this started this line used to contain the word 'unmanaged' somewhere.  Eugen and I were having a side discussion about how to make all of my OSDs managed without destroying them, so I could do things like 'ceph orch restart osd' to restart all of the OSDs to assure that the pick up changes to attributes like osd_memory_target and osd_memory_target_autotune,

So, in applying this spec, did I make all my OSDs managed, or just all of the ones on ceph01, or just the one that got created when I applied the spec?

When I add my next host, should I change the placement to that host name or to '*'?

More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Fri, Nov 1, 2024 at 1:40 PM Tim Holloway <timh@mousetech.com> wrote:

...

I can't offer a spec off the cuff, but if the LV still exists and you don't need to change its size, then I'd zap it to remove residual Ceph info because otherwise the operation will complain and fail.

Having done that, the requirements should be the same as a first- time construction of an OSD on that LV. Eugen can likely give you the spec info. I'd have to RTFM.

    Tim

On 11/1/24 11:22, Dave Hall wrote:

...

Tim, Eugen,

So what would a spec file look like for a single OSD that uses a specific HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe drive?  Regarding the NVMe, there are 3 other OSDs already using 25% each of this NVMe for WAL/DB, but I have removed the LV that was used by the failed OSD.  Do I need to pre-create the LV, or will 'ceph orch' do that for me?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@mousetech.com> wrote:

...

I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy.

Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me.

But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take.

Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes,

I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.

     Tim

On 10/31/24 15:39, Eugen Block wrote:

...

I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.

It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.

Zitat von Tim Holloway <timh@mousetech.com>:

...

I have been slowly migrating towards spec files as I prefer declarative management as a rule.

However, I think that we may have a dichotomy in the user base.

On the one hand, users with dozens/hundreds of server/drives of basically identical character.

On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage.

Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where.

At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.

    Tim

On 10/31/24 14:28, Eugen Block wrote: > Hi, > > the preferred method to deploy OSDs in cephadm managed > clusters are > spec files, see this part of the docs [0] for more > information. I > would just not use the '--all-available-devices' flag, > except in > test clusters, or if you're really sure that this is what > you want. > > If you use 'ceph orch daemon add osd ...', you'll end up > with one > (or more) OSD(s), but they will be unmanaged, as you > already noted > in your own cluster. There are a couple of examples with > advanced > specs (e. g. DB/WAL on dedicated devices) in the docs as > well [1]. > So my recommendation would be to have a suiting spec file > for your > disk layout. You can always check with the '--dry-run' > flag before > actually applying it: > > ceph orch apply -i osd-spec.yaml --dry-run > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds > [1] >

https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

...

...

...

> Zitat von Tim Holloway <timh@mousetech.com>: > > > As I understand it, the manual OSD setup is only for > > legacy > > (non-container) OSDs. Directory locations are wrong for > > managed > > (containerized) OSDs, for one. > > > > Actually, the whole manual setup docs ought to be moved > > out of the > > mainline documentation. In their present arrangement, > > they make > > legacy setup sound like the preferred method. And have > > you noticed > > that there is no corresponding well-marked section > > titled > > "Authomated (cephadmin) setup?". > > > > This is how we end up with OSDs that are simultaneously > > legacy AND > > administered for the same OSD, since at last count > > there are no > > interlocks within Ceph to prevent such a mess. > > > >     Tim > > > > On 10/31/24 13:39, Dave Hall wrote: > > > Hello. > > > > > > Sorry if it appears that I am reposting the same > > > issue under a > > > different > > > topic.  However, I feel that the problem has moved > > > and I now have > > > different > > > questions. > > > > > > At this point I have, I believe, removed all traces > > > of OSD.12 from my > > > cluster - based on steps in the Reef docs at > > > https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/# > > > . > > > I have > > > further located and removed the WAL/DB LV on an > > > associated NVMe drive > > > (shared with 3 other OSDs). > > > > > > I don't believe the instructions for replacing an OSD > > > (ceph-volume > > > lvm > > > prepare) still apply, so I have been trying to work > > > with the > > > instructions > > > under ADDING AN OSD (MANUAL). > > > > > > However, since my installation is containerized > > > (Podman), it is > > > unclear > > > which steps should be issued on the host and which > > > within 'cephadm > > > shell'. > > > > > > There is also another ambiguity:  In step 3 the > > > instruction is to > > > 'mkfs -t > > > {fstype}' and then to 'mount -o user_xattr'.  > > > However, which fs type? > > > > > > After this, in step 4, the 'ceph-osd -i {osd-id} -- > > > mkfs --mkkey' gets > > > throws errors about the keyring file. > > > > > > So, are these the right instructions to be using in a > > > containerized > > > installation?  Are there, in general, alternate > > > documents for > > > containerized > > > installations? > > > > > > Lastly, the above cited instructions don't say > > > anything about the > > > separate > > > WAL/DB LV. > > > > > > Please advise. > > > > > > Thanks. > > > > > > -Dave > > > > > > -- > > > Dave Hall > > > Binghamton University > > > kdhall@binghamton.edu > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to > > > ceph-users-leave@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to > > ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Dave, I noticed that the advanced osd spec docs are missing a link to placement-by-pattern-matching docs (thanks to Zac and Adam for picking that up): https://docs.ceph.com/en/latest/cephadm/services/#placement-by-pattern-match... But according to that, your host_pattern specification should have worked as well and target only the one node. But you don't have 32 OSDs running on ceph01, correct? I wonder if that is a bug. Do you still have the cephadm and mgr logs from when you applied that spec? I tried to check across a couple of versions from Quincy to Squid if I could reproduce it, but I couldn't. Across all versions the specified pattern did only apply to the one node I had there. Zitat von Eugen Block <eblock@nde.ag>:

Hi,

if you choose "host_pattern", it will try to match the blob as a regexp, that's why you probably see what you currently see. If you want the spec to be only applicable to a single node, you need to specify only "hosts" and then add a list (or a single entry), for example:

service_type: osd service_name: osd placement: hosts: - ceph01 spec: data_devices: rotational: 1 db_devices: rotational: 0

Zitat von Dave Hall <kdhall@binghamton.edu>:

...

Tim,

Actually, the links the Eugen shared earlier were sufficient. I ended up with

service_type: osd service_name: osd placement: host_pattern: 'ceph01' spec: data_devices: rotational: 1 db_devices: rotational: 0

This worked exactly right as far as creating the OSD - it found and reused the same OSD number that was previously destroyed, and also recreated the WAL/DB LV using the 'blank spot' on the NVMe drive.

However, I'm a bit concerned that the output of 'ceph orch ls osd' has changed in a way that might not be quite good:

NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 32 3m ago 52m ceph01

Before all of this started this line used to contain the word 'unmanaged' somewhere. Eugen and I were having a side discussion about how to make all of my OSDs managed without destroying them, so I could do things like 'ceph orch restart osd' to restart all of the OSDs to assure that the pick up changes to attributes like osd_memory_target and osd_memory_target_autotune,

So, in applying this spec, did I make all my OSDs managed, or just all of the ones on ceph01, or just the one that got created when I applied the spec?

When I add my next host, should I change the placement to that host name or to '*'?

More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Fri, Nov 1, 2024 at 1:40 PM Tim Holloway <timh@mousetech.com> wrote:

...

I can't offer a spec off the cuff, but if the LV still exists and you don't need to change its size, then I'd zap it to remove residual Ceph info because otherwise the operation will complain and fail.

Having done that, the requirements should be the same as a first-time construction of an OSD on that LV. Eugen can likely give you the spec info. I'd have to RTFM.

Tim

On 11/1/24 11:22, Dave Hall wrote:

...

Tim, Eugen,

So what would a spec file look like for a single OSD that uses a specific HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe drive? Regarding the NVMe, there are 3 other OSDs already using 25% each of this NVMe for WAL/DB, but I have removed the LV that was used by the failed OSD. Do I need to pre-create the LV, or will 'ceph orch' do that for me?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@mousetech.com> wrote:

...

I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy.

Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me.

But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take.

Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes,

I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.

Tim

On 10/31/24 15:39, Eugen Block wrote:

...

I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.

It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.

Zitat von Tim Holloway <timh@mousetech.com>:

> I have been slowly migrating towards spec files as I prefer > declarative management as a rule. > > However, I think that we may have a dichotomy in the user base. > > On the one hand, users with dozens/hundreds of server/drives of > basically identical character. > > On the other, I'm one who's running fewer servers and for historical > reasons they tend to be wildly individualistic and often have blocks > of future-use space reserved for non-ceph storage. > > Ceph, left to its own devices (no pun intended) can be quite > enthusiastic about adopting any storage it can find. And that's great > for users in the first category. Which is what the spec information > in the supplied links is emphasizing. But for us lesser creatures who > feel the need to manually control where each OSD and how it's > configured, it's not so simple. I'm fairly certain that there's > documentation on the spec file setup for that sort of stuff in the > online docs, but it's located somewhere else and I cannot recall where. > > At any rate I would consider it very important that the different > ways to setup an OSD should explicitly indicate which type of OSD > will be generated in their documentation. > > Tim > > > On 10/31/24 14:28, Eugen Block wrote: >> Hi, >> >> the preferred method to deploy OSDs in cephadm managed clusters are >> spec files, see this part of the docs [0] for more information. I >> would just not use the '--all-available-devices' flag, except in >> test clusters, or if you're really sure that this is what you want. >> >> If you use 'ceph orch daemon add osd ...', you'll end up with one >> (or more) OSD(s), but they will be unmanaged, as you already noted >> in your own cluster. There are a couple of examples with advanced >> specs (e. g. DB/WAL on dedicated devices) in the docs as well [1]. >> So my recommendation would be to have a suiting spec file for your >> disk layout. You can always check with the '--dry-run' flag before >> actually applying it: >> >> ceph orch apply -i osd-spec.yaml --dry-run >> >> Regards, >> Eugen >> >> [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds >> [1] >>

https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

...

...

>> Zitat von Tim Holloway <timh@mousetech.com>: >> >>> As I understand it, the manual OSD setup is only for legacy >>> (non-container) OSDs. Directory locations are wrong for managed >>> (containerized) OSDs, for one. >>> >>> Actually, the whole manual setup docs ought to be moved out of the >>> mainline documentation. In their present arrangement, they make >>> legacy setup sound like the preferred method. And have you noticed >>> that there is no corresponding well-marked section titled >>> "Authomated (cephadmin) setup?". >>> >>> This is how we end up with OSDs that are simultaneously legacy AND >>> administered for the same OSD, since at last count there are no >>> interlocks within Ceph to prevent such a mess. >>> >>> Tim >>> >>> On 10/31/24 13:39, Dave Hall wrote: >>>> Hello. >>>> >>>> Sorry if it appears that I am reposting the same issue under a >>>> different >>>> topic. However, I feel that the problem has moved and I now have >>>> different >>>> questions. >>>> >>>> At this point I have, I believe, removed all traces of OSD.12 from my >>>> cluster - based on steps in the Reef docs at >>>> https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. >>>> I have >>>> further located and removed the WAL/DB LV on an associated NVMe drive >>>> (shared with 3 other OSDs). >>>> >>>> I don't believe the instructions for replacing an OSD (ceph-volume >>>> lvm >>>> prepare) still apply, so I have been trying to work with the >>>> instructions >>>> under ADDING AN OSD (MANUAL). >>>> >>>> However, since my installation is containerized (Podman), it is >>>> unclear >>>> which steps should be issued on the host and which within 'cephadm >>>> shell'. >>>> >>>> There is also another ambiguity: In step 3 the instruction is to >>>> 'mkfs -t >>>> {fstype}' and then to 'mount -o user_xattr'. However, which fs type? >>>> >>>> After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets >>>> throws errors about the keyring file. >>>> >>>> So, are these the right instructions to be using in a containerized >>>> installation? Are there, in general, alternate documents for >>>> containerized >>>> installations? >>>> >>>> Lastly, the above cited instructions don't say anything about the >>>> separate >>>> WAL/DB LV. >>>> >>>> Please advise. >>>> >>>> Thanks. >>>> >>>> -Dave >>>> >>>> -- >>>> Dave Hall >>>> Binghamton University >>>> kdhall@binghamton.edu >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

----- Le 1 Nov 24, à 19:28, Dave Hall kdhall@binghamton.edu a écrit :

Tim,

Actually, the links the Eugen shared earlier were sufficient. I ended up with

service_type: osd service_name: osd placement: host_pattern: 'ceph01' spec: data_devices: rotational: 1 db_devices: rotational: 0

This worked exactly right as far as creating the OSD - it found and reused the same OSD number that was previously destroyed, and also recreated the WAL/DB LV using the 'blank spot' on the NVMe drive.

However, I'm a bit concerned that the output of 'ceph orch ls osd' has changed in a way that might not be quite good:

NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 32 3m ago 52m ceph01

Hi Dave, Was this cluster cephadm adopted? That is moved from non-containerized to containerized cluster. This would explain why you have these 32 OSDs bound to this unmanaged and actually non-existing (I just figured that out...) 'osd' service. This service appears in 'ceph orch ls' output as soon as an OSD is converted to a container and restarted with a service_name set to osd.osd in its systemd unit.meta file. But this service doesn't actually exist, in the sens that you cannot remove it (hence the ceph orch rm issue we discussed here a couple of weeks back). You could use this service but I would avoid doing so as it doesn't really exist (here may be a bug) and will disappear as soon as no more OSDs reference it.

Before all of this started this line used to contain the word 'unmanaged' somewhere. Eugen and I were having a side discussion about how to make all of my OSDs managed without destroying them, so I could do things like 'ceph orch restart osd' to restart all of the OSDs to assure that the pick up changes to attributes like osd_memory_target and osd_memory_target_autotune,

So, in applying this spec, did I make all my OSDs managed, or just all of the ones on ceph01, or just the one that got created when I applied the spec?

By applying this spec, you've asked the orchestrator to create OSDs out of any available disks it will find on host ceph01 **only**. The orchestrator will not try to create any OSDs on any other hosts after you limit the scope of the service to this particular host (host_pattern actually). Note that this reduced scope doesn't prevent the orchestrator from restarting all OSDs would you run a 'ceph orch restart osd.osd'.

When I add my next host, should I change the placement to that host name or to '*'?

You could enumerate all hosts one by one or use a pattern like 'ceph0[1-2]' You may also use regex patterns depending on the version of Ceph that you're using. Check [1]. Regex patterns should be available in next minor Quincy release 17.2.8. [1] https://github.com/ceph/ceph/pull/53803 [2] https://github.com/ceph/ceph/pull/56222

More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?

I think Eugen got you on track. :-) Cheers, Frédéric.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Fri, Nov 1, 2024 at 1:40 PM Tim Holloway <timh@mousetech.com> wrote:

...

I can't offer a spec off the cuff, but if the LV still exists and you don't need to change its size, then I'd zap it to remove residual Ceph info because otherwise the operation will complain and fail.

Having done that, the requirements should be the same as a first-time construction of an OSD on that LV. Eugen can likely give you the spec info. I'd have to RTFM.

Tim

On 11/1/24 11:22, Dave Hall wrote:

...

Tim, Eugen,

So what would a spec file look like for a single OSD that uses a specific HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe drive? Regarding the NVMe, there are 3 other OSDs already using 25% each of this NVMe for WAL/DB, but I have removed the LV that was used by the failed OSD. Do I need to pre-create the LV, or will 'ceph orch' do that for me?

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu

On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@mousetech.com> wrote:

...

I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy.

Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me.

But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take.

Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes,

I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.

Tim

On 10/31/24 15:39, Eugen Block wrote:

...

I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.

It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.

Zitat von Tim Holloway <timh@mousetech.com>:

...

I have been slowly migrating towards spec files as I prefer declarative management as a rule.

However, I think that we may have a dichotomy in the user base.

On the one hand, users with dozens/hundreds of server/drives of basically identical character.

On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage.

Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where.

At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.

Tim

On 10/31/24 14:28, Eugen Block wrote: > Hi, > > the preferred method to deploy OSDs in cephadm managed clusters are > spec files, see this part of the docs [0] for more information. I > would just not use the '--all-available-devices' flag, except in > test clusters, or if you're really sure that this is what you want. > > If you use 'ceph orch daemon add osd ...', you'll end up with one > (or more) OSD(s), but they will be unmanaged, as you already noted > in your own cluster. There are a couple of examples with advanced > specs (e. g. DB/WAL on dedicated devices) in the docs as well [1]. > So my recommendation would be to have a suiting spec file for your > disk layout. You can always check with the '--dry-run' flag before > actually applying it: > > ceph orch apply -i osd-spec.yaml --dry-run > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds > [1] >

https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...

...

...

...

> Zitat von Tim Holloway <timh@mousetech.com>: > >> As I understand it, the manual OSD setup is only for legacy >> (non-container) OSDs. Directory locations are wrong for managed >> (containerized) OSDs, for one. >> >> Actually, the whole manual setup docs ought to be moved out of the >> mainline documentation. In their present arrangement, they make >> legacy setup sound like the preferred method. And have you noticed >> that there is no corresponding well-marked section titled >> "Authomated (cephadmin) setup?". >> >> This is how we end up with OSDs that are simultaneously legacy AND >> administered for the same OSD, since at last count there are no >> interlocks within Ceph to prevent such a mess. >> >> Tim >> >> On 10/31/24 13:39, Dave Hall wrote: >>> Hello. >>> >>> Sorry if it appears that I am reposting the same issue under a >>> different >>> topic. However, I feel that the problem has moved and I now have >>> different >>> questions. >>> >>> At this point I have, I believe, removed all traces of OSD.12 from my >>> cluster - based on steps in the Reef docs at >>> https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. >>> I have >>> further located and removed the WAL/DB LV on an associated NVMe drive >>> (shared with 3 other OSDs). >>> >>> I don't believe the instructions for replacing an OSD (ceph-volume >>> lvm >>> prepare) still apply, so I have been trying to work with the >>> instructions >>> under ADDING AN OSD (MANUAL). >>> >>> However, since my installation is containerized (Podman), it is >>> unclear >>> which steps should be issued on the host and which within 'cephadm >>> shell'. >>> >>> There is also another ambiguity: In step 3 the instruction is to >>> 'mkfs -t >>> {fstype}' and then to 'mount -o user_xattr'. However, which fs type? >>> >>> After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets >>> throws errors about the keyring file. >>> >>> So, are these the right instructions to be using in a containerized >>> installation? Are there, in general, alternate documents for >>> containerized >>> installations? >>> >>> Lastly, the above cited instructions don't say anything about the >>> separate >>> WAL/DB LV. >>> >>> Please advise. >>> >>> Thanks. >>> >>> -Dave >>> >>> -- >>> Dave Hall >>> Binghamton University >>> kdhall@binghamton.edu >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Quick comment on the CLI argos vs. the spec file. It actually shouldn't allow you to do both for any flags that actually affect the service. If you run `ceph orch apply -i <filepath>` it will only make use of the spec file and should return an error if flags that affect the service like `--unamanged` or `--placement` are passed. Separately, there are a number of `ceph orch apply <service-name>` commands e.g. `ceph orch apply nfs` that will return an error if a spec file is passed in. The only exception off the top of my head is `ceph orch apply osd` which will force you to either pass a spec file or pass `--all-available-devices`. If anything else does let you pass in relevant CLI args and a spec file at the same time it's likely an oversight / bug. As for the osd service, it is correct that the service just called "osd" is not a real service / a placeholder for OSDs that don't have a service. In later releases it will actually block you from creating a service with service_type: osd service_name: osd so that that placeholder isn't attached to an actual spec file. Since that's already happened, what I'd personally recommend is re-applying that spec file but without the `service_name` field and instead adding something like `service_id: foo`. That would create an osd.foo service. For the time being, given nothing else in the spec was modified, it won't have any OSDs on it as the only matching device has already been made part of an OSD. To get that OSD to fall under that service, you can go onto the `ceph01` node directly, and edit the "service_name" field of the /var/lib/ceph/<fsid>/osd.<osd-id>/unit.meta file to be "osd.foo" or whatever name you pick, then run `ceph orch ps --refresh`, wait a bit and it should appear under `osd.foo` rather than the "osd" service. Then you can just do `ceph orch rm osd` to delete the old spec file that became attached to the "osd" service. In the future there are also plans to add a `ceph orch` command to change the OSD spec affinity so you don't need to do the extra steps, as well as probably allowing service actions against the "osd" service (even though it's just a placeholder in reality) but none of that exists currently. On Wed, Nov 6, 2024 at 11:50 AM Tim Holloway <timh@mousetech.com> wrote:

On 11/6/24 11:04, Frédéric Nass wrote:

...

... You could enumerate all hosts one by one or use a pattern like 'ceph0[1-2]'

You may also use regex patterns depending on the version of Ceph that you're using. Check [1]. Regex patterns should be available in next minor Quincy release 17.2.8.

[1] https://github.com/ceph/ceph/pull/53803 [2] https://github.com/ceph/ceph/pull/56222

This simple wildcard scheme is called "globbing" and it's what is typically used by Unix-style command shells.

It's convenient, but limited. For more subtlety you'd need a regex.

...

...

More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?

I've submitted an outline on the issue tracker for a comprehensive and coherent document on the creation and maintenance of OSDs.

Spec files, as I mentioned earlier, are YAML-format declarations of the specifications to be applied to create or update one or more resources. You can define spec files not only for OSDs, but for other Ceph resources as well and I *think* a unified spec file for multiple resource types.

By convention, any orchestrator command that can get its options from a spec file will do so via the "-i {filename}" command line option. If you put options on both the spec file and the orchestrator command line, presumably the command line will take precedence, as that's the general convention for such things. It may or may not warn you that it's doing so, though.

Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Adam, Thanks a lot for sharing this information. It confirms our observations. Regards, Frédéric. ----- Le 6 Nov 24, à 20:23, Adam King adking@redhat.com a écrit :

Quick comment on the CLI argos vs. the spec file. It actually shouldn't allow you to do both for any flags that actually affect the service. If you run `ceph orch apply -i <filepath>` it will only make use of the spec file and should return an error if flags that affect the service like `--unamanged` or `--placement` are passed. Separately, there are a number of `ceph orch apply <service-name>` commands e.g. `ceph orch apply nfs` that will return an error if a spec file is passed in. The only exception off the top of my head is `ceph orch apply osd` which will force you to either pass a spec file or pass `--all-available-devices`. If anything else does let you pass in relevant CLI args and a spec file at the same time it's likely an oversight / bug.

As for the osd service, it is correct that the service just called "osd" is not a real service / a placeholder for OSDs that don't have a service. In later releases it will actually block you from creating a service with

service_type: osd service_name: osd

so that that placeholder isn't attached to an actual spec file. Since that's already happened, what I'd personally recommend is re-applying that spec file but without the `service_name` field and instead adding something like `service_id: foo`. That would create an osd.foo service. For the time being, given nothing else in the spec was modified, it won't have any OSDs on it as the only matching device has already been made part of an OSD. To get that OSD to fall under that service, you can go onto the `ceph01` node directly, and edit the "service_name" field of the /var/lib/ceph/<fsid>/osd.<osd-id>/unit.meta file to be "osd.foo" or whatever name you pick, then run `ceph orch ps --refresh`, wait a bit and it should appear under `osd.foo` rather than the "osd" service. Then you can just do `ceph orch rm osd` to delete the old spec file that became attached to the "osd" service. In the future there are also plans to add a `ceph orch` command to change the OSD spec affinity so you don't need to do the extra steps, as well as probably allowing service actions against the "osd" service (even though it's just a placeholder in reality) but none of that exists currently.

On Wed, Nov 6, 2024 at 11:50 AM Tim Holloway <timh@mousetech.com> wrote:

...

On 11/6/24 11:04, Frédéric Nass wrote:

...

... You could enumerate all hosts one by one or use a pattern like 'ceph0[1-2]'

You may also use regex patterns depending on the version of Ceph that you're using. Check [1]. Regex patterns should be available in next minor Quincy release 17.2.8.

[1] https://github.com/ceph/ceph/pull/53803 [2] https://github.com/ceph/ceph/pull/56222

This simple wildcard scheme is called "globbing" and it's what is typically used by Unix-style command shells.

It's convenient, but limited. For more subtlety you'd need a regex.

...

...

More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?

I've submitted an outline on the issue tracker for a comprehensive and coherent document on the creation and maintenance of OSDs.

Spec files, as I mentioned earlier, are YAML-format declarations of the specifications to be applied to create or update one or more resources. You can define spec files not only for OSDs, but for other Ceph resources as well and I *think* a unified spec file for multiple resource types.

By convention, any orchestrator command that can get its options from a spec file will do so via the "-i {filename}" command line option. If you put options on both the spec file and the orchestrator command line, presumably the command line will take precedence, as that's the general convention for such things. It may or may not warn you that it's doing so, though.

Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io