I'm afraid I've never paid much attention to "ceprh orch ls osd". Mostly I look at ceph orch ps or ceph osd tree and I nenver noticed whether they had a specific indicator for managed/unmanaged. The spec file is bog-standard YAML (Yet Another Markup Language) and the actual valid elements for a given spec are defined by its consumer. There's very good documentation on the spec info for OSDs. I think that Ceph can actually run off a master spec file with all of the options for all types of resources in it, but I cannot be certain. "*" is simply a wildcard pattern meaning "all servers". If I had, for example only certain hosts reserved for OSDs, say 'osd01.ceph.mousetech.com', then a pattern like 'osd*.ceph.mousetech.com' might work. OSDs do not automatically convert. You have to issue an explicit cephadm command to make a legacy OSD become an administered one. And it will fail if you've done something like double-define an OSD. Tim On Fri, 2024-11-01 at 14:28 -0400, Dave Hall wrote:
Tim,
Actually, the links the Eugen shared earlier were sufficient. I ended up with
service_type: osd service_name: osd placement: host_pattern: 'ceph01' spec: data_devices: rotational: 1 db_devices: rotational: 0
This worked exactly right as far as creating the OSD - it found and reused the same OSD number that was previously destroyed, and also recreated the WAL/DB LV using the 'blank spot' on the NVMe drive.
However, I'm a bit concerned that the output of 'ceph orch ls osd' has changed in a way that might not be quite good:
NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 32 3m ago 52m ceph01
Before all of this started this line used to contain the word 'unmanaged' somewhere. Eugen and I were having a side discussion about how to make all of my OSDs managed without destroying them, so I could do things like 'ceph orch restart osd' to restart all of the OSDs to assure that the pick up changes to attributes like osd_memory_target and osd_memory_target_autotune,
So, in applying this spec, did I make all my OSDs managed, or just all of the ones on ceph01, or just the one that got created when I applied the spec?
When I add my next host, should I change the placement to that host name or to '*'?
More generally, is there a higher level document that talks about Ceph spec files and the orchestrator - something that deals with the general concepts?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Fri, Nov 1, 2024 at 1:40 PM Tim Holloway <timh@mousetech.com> wrote:
I can't offer a spec off the cuff, but if the LV still exists and you don't need to change its size, then I'd zap it to remove residual Ceph info because otherwise the operation will complain and fail.
Having done that, the requirements should be the same as a first- time construction of an OSD on that LV. Eugen can likely give you the spec info. I'd have to RTFM.
Tim
On 11/1/24 11:22, Dave Hall wrote:
Tim, Eugen,
So what would a spec file look like for a single OSD that uses a specific HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe drive? Regarding the NVMe, there are 3 other OSDs already using 25% each of this NVMe for WAL/DB, but I have removed the LV that was used by the failed OSD. Do I need to pre-create the LV, or will 'ceph orch' do that for me?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@mousetech.com> wrote:
I migrated from gluster when I found out it's going unsupported shortly. I'm really not big enough for Ceph proper, but there were only so many supported distributed filesystems with triple redundancy.
Where I got into trouble was that I started off with Octopus and Octopus had some teething pains. Like stalling scheduled operations until the system was clan but the only way to get a clean system was to run the stalled operations. Pacific cured that for me.
But the docs were and remain somewhat fractured between legacy and managed services and I managed to get into a real mess there, especially since I was wildly trying anything to get those stalled fixes to take.
Since then, I've pretty much redefined all my OSDs with fewer but larger datastores and made them all managed. Now if I could just persuade the auto-tuner to fix the PG sizes,
I'm in the process of opening a ticket account right now. The fun part of this is that realistically, older docs need a re-write just as much as the docs for the current release.
Tim
On 10/31/24 15:39, Eugen Block wrote:
I completely understand your point of view. Our own main cluster is also a bit "wild" in its OSD layout, that's why its OSDs are "unmanaged" as well. When we adopted it via cephadm, I started to create suitable osd specs for all those hosts and OSDs and I gave up. :-D But since we sometimes also tend to experiment a bit, I rather have full control over it. That's why we also have osd_crush_initial_weight = 0, to check the OSD creation before letting Ceph remap any PGs.
It definitely couldn't hurt to clarify the docs, you can always report on tracker.ceph.com if you have any improvement ideas.
Zitat von Tim Holloway <timh@mousetech.com>:
I have been slowly migrating towards spec files as I prefer declarative management as a rule.
However, I think that we may have a dichotomy in the user base.
On the one hand, users with dozens/hundreds of server/drives of basically identical character.
On the other, I'm one who's running fewer servers and for historical reasons they tend to be wildly individualistic and often have blocks of future-use space reserved for non-ceph storage.
Ceph, left to its own devices (no pun intended) can be quite enthusiastic about adopting any storage it can find. And that's great for users in the first category. Which is what the spec information in the supplied links is emphasizing. But for us lesser creatures who feel the need to manually control where each OSD and how it's configured, it's not so simple. I'm fairly certain that there's documentation on the spec file setup for that sort of stuff in the online docs, but it's located somewhere else and I cannot recall where.
At any rate I would consider it very important that the different ways to setup an OSD should explicitly indicate which type of OSD will be generated in their documentation.
Tim
On 10/31/24 14:28, Eugen Block wrote: > Hi, > > the preferred method to deploy OSDs in cephadm managed > clusters are > spec files, see this part of the docs [0] for more > information. I > would just not use the '--all-available-devices' flag, > except in > test clusters, or if you're really sure that this is what > you want. > > If you use 'ceph orch daemon add osd ...', you'll end up > with one > (or more) OSD(s), but they will be unmanaged, as you > already noted > in your own cluster. There are a couple of examples with > advanced > specs (e. g. DB/WAL on dedicated devices) in the docs as > well [1]. > So my recommendation would be to have a suiting spec file > for your > disk layout. You can always check with the '--dry-run' > flag before > actually applying it: > > ceph orch apply -i osd-spec.yaml --dry-run > > Regards, > Eugen > > [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds > [1] >
https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
> Zitat von Tim Holloway <timh@mousetech.com>: > > > As I understand it, the manual OSD setup is only for > > legacy > > (non-container) OSDs. Directory locations are wrong for > > managed > > (containerized) OSDs, for one. > > > > Actually, the whole manual setup docs ought to be moved > > out of the > > mainline documentation. In their present arrangement, > > they make > > legacy setup sound like the preferred method. And have > > you noticed > > that there is no corresponding well-marked section > > titled > > "Authomated (cephadmin) setup?". > > > > This is how we end up with OSDs that are simultaneously > > legacy AND > > administered for the same OSD, since at last count > > there are no > > interlocks within Ceph to prevent such a mess. > > > > Tim > > > > On 10/31/24 13:39, Dave Hall wrote: > > > Hello. > > > > > > Sorry if it appears that I am reposting the same > > > issue under a > > > different > > > topic. However, I feel that the problem has moved > > > and I now have > > > different > > > questions. > > > > > > At this point I have, I believe, removed all traces > > > of OSD.12 from my > > > cluster - based on steps in the Reef docs at > > > https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/# > > > . > > > I have > > > further located and removed the WAL/DB LV on an > > > associated NVMe drive > > > (shared with 3 other OSDs). > > > > > > I don't believe the instructions for replacing an OSD > > > (ceph-volume > > > lvm > > > prepare) still apply, so I have been trying to work > > > with the > > > instructions > > > under ADDING AN OSD (MANUAL). > > > > > > However, since my installation is containerized > > > (Podman), it is > > > unclear > > > which steps should be issued on the host and which > > > within 'cephadm > > > shell'. > > > > > > There is also another ambiguity: In step 3 the > > > instruction is to > > > 'mkfs -t > > > {fstype}' and then to 'mount -o user_xattr'. > > > However, which fs type? > > > > > > After this, in step 4, the 'ceph-osd -i {osd-id} -- > > > mkfs --mkkey' gets > > > throws errors about the keyring file. > > > > > > So, are these the right instructions to be using in a > > > containerized > > > installation? Are there, in general, alternate > > > documents for > > > containerized > > > installations? > > > > > > Lastly, the above cited instructions don't say > > > anything about the > > > separate > > > WAL/DB LV. > > > > > > Please advise. > > > > > > Thanks. > > > > > > -Dave > > > > > > -- > > > Dave Hall > > > Binghamton University > > > kdhall@binghamton.edu > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to > > > ceph-users-leave@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to > > ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io