Accidentally created systemd units for OSDs
I was [poorly] following the instructions for migrating the wal/db to an SSD https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for... and I didn't add the '--no-systemd' when I did 'ceph-volume lvm activate' command (3 f***ing times). The result is that I've "twinned" 3 of my OSDs: There's a container version managed by cephadm and there's an instantiated systemd unit that runs directly. Surprisingly, this has not done a lot of damage, but it does result in the dashboard reporting 3 failed cephadm daemons when the "native" OSDs start before the containerized ones. I've disabled the systemd units for ceph-osd@9.service, ceph-osd@11.service and ceph-osd@25.service, but I'd like to remove them completely. I will eventually badger The Google into giving me an answer, but could someone tell me what I need to do? The semester starts soon and I don't really have the bandwidth for this right now. Thanks in advance. I will forever be in your debt. (Seriously, I'm ready to give you a kidney, if you need it.)
Been there/did that. Cried a lot. Fixed now. Personally, I recommend the containerise/cephadm-managed approach. In a lot of ways, it's simpler and it supports more than one fsid on a single host.The downside is that the systemd names are really gnarly (the full fsid is part of the unitname) and they're generated on-the- fly so you won't find the unitnames in any of the systemd static directories (/usr/lib/systemd/systemd and /etc/systemd/system). As I recall, having 2 different managers for an OSD didn't hurt overall operation that much, but it did make status reporting somewhat flakey. Apparently different tools check OSDs differently. The other problem is that the traditional (non-container) OSD definitions are located under /var.lib/ceph, but the containerized OSDs are under /var/lib/ceph/<fsid>. There are some softlinks involved so that, for example, the OSD data itself may be shared between the two OSD controllers, and that has the potential for conflict. I think that the safest way to handle that is to completely nuke that OSD in both its forms, then re-create it according to your mode of choice. That can take a long time, but shortcuts may make things worse. Pity that Ceph can't detect and prevent that sort of stuff, but such is life. Tim On Fri, 2024-08-16 at 15:32 +0000, Dan O'Brien wrote:
I was [poorly] following the instructions for migrating the wal/db to an SSD
https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-for...
and I didn't add the '--no-systemd' when I did 'ceph-volume lvm activate' command (3 f***ing times). The result is that I've "twinned" 3 of my OSDs: There's a container version managed by cephadm and there's an instantiated systemd unit that runs directly. Surprisingly, this has not done a lot of damage, but it does result in the dashboard reporting 3 failed cephadm daemons when the "native" OSDs start before the containerized ones.
I've disabled the systemd units for ceph-osd@9.service, ceph-osd@11.service and ceph-osd@25.service, but I'd like to remove them completely. I will eventually badger The Google into giving me an answer, but could someone tell me what I need to do? The semester starts soon and I don't really have the bandwidth for this right now.
Thanks in advance. I will forever be in your debt. (Seriously, I'm ready to give you a kidney, if you need it.) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I am 100% using cephadm and containers and plan to continue to do so. Our original setup was all spinners, but after going to Ceph Days NYC, I pushed for SSDs to use for the WAL/RocksDb and I'm in the process of migrating the WAL/RocksDb. In general, it's been fairly straightforward -- IF YOU FOLLOW THE DIRECTIONS EXACTLY. From what I can tell (and someone please correct me if I'm wrong), it appears that I've just introduced a bit of cruft into systemd that duplicates the configuration of the container OSD. If I can get rid of that bit of bellybutton-lint, I think I'm OK without rebuilding the OSD. (At least until I screw it up again on one of the remaining OSDs I need to migrate). Anyone know how to get rid of an instance of a templated systemd unit? PLEEEEEEEEEEZE?
I would expect it to be: systemctl disable ceph-osd@${instance} If you're wanting to disable them all I believe you can even use wildcards: systemctl disable ceph-osd@\* -- Adam On 8/16/24 2:24 PM, Dan O'Brien wrote:
This email originated from outside of K-State.
I am 100% using cephadm and containers and plan to continue to do so.
Our original setup was all spinners, but after going to Ceph Days NYC, I pushed for SSDs to use for the WAL/RocksDb and I'm in the process of migrating the WAL/RocksDb. In general, it's been fairly straightforward -- IF YOU FOLLOW THE DIRECTIONS EXACTLY. From what I can tell (and someone please correct me if I'm wrong), it appears that I've just introduced a bit of cruft into systemd that duplicates the configuration of the container OSD. If I can get rid of that bit of bellybutton-lint, I think I'm OK without rebuilding the OSD. (At least until I screw it up again on one of the remaining OSDs I need to migrate).
Anyone know how to get rid of an instance of a templated systemd unit? PLEEEEEEEEEEZE? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If it makes you feel better, that sounds exactly like what happened to me and I have no idea how. Other than I'd started with Octopus and it was a transitional release, there are conflicting instructions AND a reference in the Octopus docs to procedures using a tool that was no longer distributed with Octopus. If by "templated" you mean what I think you mean, the cleanup I'd recommend is as follows: 1. Follow documented processed for draining and deleting an OSD (the dashboard seems to handle this well). 2. Make sure that your non-container service is shut down. 3. Also make sure that your OSD is completely removed from the "osd tree" list. That may require a manual reweighting to 0 which I did from the command line. My thanks to EEugen Block for that assist. 4. At this point, hopefully a "systemctl status" won't show either the old-style or container-style OSD services as active. In which case, you may manually delete the specific OSD unit file from /etc/systemd/system and it probably won't hurt if you delete the template, but I'd recommend leaving it, since it will just come back soon. 5. finally, completely erase the OSD directory under /var/lib/ceph and its counterpart under /var/lib/ceph/<fsid>, If they're already gone, so much the better. The template OSD is used by "ceph control" to dynamically generate the systemd unit file under /run/ceph/<fsid> when it detects an OSD definition. Which is probably keyed by finding a resource under /var/lib/ceph/<fsid>/osd.x, if not actually the OSD directory itself. Unlike /etc/systemd and /etc/systemd, the /run/systemd is effectively destroyed when the system is down and its ceph files aren't permanent. Manually erasing them thus has no benefit. So, in other words, the templated OSD systemd unit won't get created unless the system actually has cephadm OSDs defined and your problem vanishes when the OSDs do! If it's of any help, I've a quick-and-dirty way to spawn minimal VMs and OSDs you can pull from https://gogs.mousetech.com. It has 2 parts. One to construct a VM, the other is an Ansible playbooks for installing the necessary cephadm infrastructure on a new machine, Which should work on bare OS's as well, as far as I can see. Though I've not tested that. Customize and/or plunder for whatever benefits it might give you. Tim On Fri, 2024-08-16 at 19:24 +0000, Dan O'Brien wrote:
I am 100% using cephadm and containers and plan to continue to do so.
Our original setup was all spinners, but after going to Ceph Days NYC, I pushed for SSDs to use for the WAL/RocksDb and I'm in the process of migrating the WAL/RocksDb. In general, it's been fairly straightforward -- IF YOU FOLLOW THE DIRECTIONS EXACTLY. From what I can tell (and someone please correct me if I'm wrong), it appears that I've just introduced a bit of cruft into systemd that duplicates the configuration of the container OSD. If I can get rid of that bit of bellybutton-lint, I think I'm OK without rebuilding the OSD. (At least until I screw it up again on one of the remaining OSDs I need to migrate).
Anyone know how to get rid of an instance of a templated systemd unit? PLEEEEEEEEEEZE? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK... I've been in the Circle of Hell where systemd lives and I *THINK* I have convinced myself I'm OK. I *REALLY* don't want to trash and rebuild the OSDs. In the manpage for systemd.unit, I found UNIT GARBAGE COLLECTION The system and service manager loads a unit's configuration automatically when a unit is referenced for the first time. It will automatically unload the unit configuration and state again when the unit is not needed anymore ("garbage collection"). I've disabled the systemd units (which removes the symlink from the target) for the non-cephadm OSDs I created by mistake and I'm PRETTY SURE if I wait long enough (or reboot) that I won't see them any more, since there won't be a unit for systemd to care about. I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just now, but it says "device busy." I think that's because there's some OTHER systemd cruft that shows a mount: [root@ceph02 ~]# systemctl --all | grep ceph | grep mount var-lib-ceph-osd-ceph\x2d11.mount loaded active mounted /var/lib/ceph/osd/ceph-11 var-lib-ceph-osd-ceph\x2d25.mount loaded active mounted /var/lib/ceph/osd/ceph-25 var-lib-ceph-osd-ceph\x2d9.mount loaded active mounted /var/lib/ceph/osd/ceph-9 When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish.
It depends on your available resources, but I really do recommend destroying and re-creating that OSD. If you have to spin up a VM and set up a temporary OSD just to keep the overall system happy, even that is a small price to pay. As I said, you can't unlink/disable the container systemd, because it's created and re-recreated dynamically. The non-container systemd unit is easier to deal with, since it's static but if you're sharing resources like I was, I still think it safer if you migrate everything out of that OSD, totally destroy all traces of both realizations of it, then re-create and migrate back. Ceph can do that, and although it can take a while to do the migration, it's well worth the effort, Ceph is very good at keeping data safe, but I'd not abuse the privilege. Oh yeah, you have multiple OSDs with that problem. Do one at a time and you'll go faster, Doing a mass shovel of data in and out of one OSD takes long enough, but trying to run the process in parallel may well bog down the whole Ceph network. As I recall, as part of my own fumbling approach, I first stopped and masked the osd-style systemd unit. But as I said, different subsystems look in different places to monitor OSDs and I'd get conflicting status reports. So while in theory, you could brute-force remove the systemd unit and the /car/lib/ceph/osd.x directory subtree (preferably while the container is also offline, I think it cleaner and safer overall to simply go back to a blank slate. On Fri, 2024-08-16 at 21:05 +0000, Dan O'Brien wrote:
OK... I've been in the Circle of Hell where systemd lives and I *THINK* I have convinced myself I'm OK. I *REALLY* don't want to trash and rebuild the OSDs.
In the manpage for systemd.unit, I found
UNIT GARBAGE COLLECTION The system and service manager loads a unit's configuration automatically when a unit is referenced for the first time. It will automatically unload the unit configuration and state again when the unit is not needed anymore ("garbage collection").
I've disabled the systemd units (which removes the symlink from the target) for the non-cephadm OSDs I created by mistake and I'm PRETTY SURE if I wait long enough (or reboot) that I won't see them any more, since there won't be a unit for systemd to care about.
I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just now, but it says "device busy." I think that's because there's some OTHER systemd cruft that shows a mount: [root@ceph02 ~]# systemctl --all | grep ceph | grep mount var-lib-ceph-osd-ceph\x2d11.mount loaded active mounted /var/lib/ceph/osd/ceph-11 var-lib-ceph-osd-ceph\x2d25.mount loaded active mounted /var/lib/ceph/osd/ceph-25 var-lib-ceph-osd-ceph\x2d9.mount loaded active mounted /var/lib/ceph/osd/ceph-9
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish.
note that there are still users/operators/admins who don't use containers. Changing the ceph-volume default might not be the best idea in this case. Regarding the cleanup, this was the thread [1] Tim was referring to. I would set the noout flag, stop an OSD (so the device won't be busy anymore), make sure that both ceph-osd@{OSD_ID} and ceph-{FSID}@osd.{OSD_ID} then double check that everything you need is still under /var/lib/ceph/{FSID}/osd.{OSD_ID}, like configs an keyrings. Disable the ceph-osd@{OSD_ID} (as already pointed out), then check if the orchestrator can start the OSD via systemd: ceph orch daemon start osd.{OSD_ID} or alternatively, try it manually: systemctl reset-failed systemctl start ceph-{FSID}@osd.{OSD_ID} Watch the log for that OSD to identify any issues. If it works, unset the noout flag. You might want to ensure it also works after a reboot, though. I don't think it should be necessary to redeploy the OSDs, but the cleanup has to be proper. As a guidance you can check the cephadm tool's contents and look for the "adopt" function. That migrates the contents of the pre-cephadm daemons into the FSID specific directories. Regards, Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/K2R3MXRD3S2... Zitat von Dan O'Brien <dobrie2@gmu.edu>:
OK... I've been in the Circle of Hell where systemd lives and I *THINK* I have convinced myself I'm OK. I *REALLY* don't want to trash and rebuild the OSDs.
In the manpage for systemd.unit, I found
UNIT GARBAGE COLLECTION The system and service manager loads a unit's configuration automatically when a unit is referenced for the first time. It will automatically unload the unit configuration and state again when the unit is not needed anymore ("garbage collection").
I've disabled the systemd units (which removes the symlink from the target) for the non-cephadm OSDs I created by mistake and I'm PRETTY SURE if I wait long enough (or reboot) that I won't see them any more, since there won't be a unit for systemd to care about.
I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just now, but it says "device busy." I think that's because there's some OTHER systemd cruft that shows a mount: [root@ceph02 ~]# systemctl --all | grep ceph | grep mount var-lib-ceph-osd-ceph\x2d11.mount loaded active mounted /var/lib/ceph/osd/ceph-11 var-lib-ceph-osd-ceph\x2d25.mount loaded active mounted /var/lib/ceph/osd/ceph-25 var-lib-ceph-osd-ceph\x2d9.mount loaded active mounted /var/lib/ceph/osd/ceph-9
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'd put in an RFO to detect/prevent creation of mutually-exclusive OSD definitions on a single OSD storage unit myself, since that's the real problem. As Eugen has noted, you can up-convert a traditional OSD to cephadm manaegment... unless there's already a managed instance existing. I can attest that from experience. the reasons I recommend complete destruction/rebuilding of the offending OSDs boil down to this: 1. Not knowing the internals of either new or old OSD logic, I feel it risky to just rip things out by brute force. 2. Because two independent and mutually-ignorant processes have been working on the same OSD backing store, there is possibility for corruption.I don't expect an OSD to have suitable interlocks to prevent that, since normally a single OSD is the sole owner of its backing store and interlocks would just slow it down for no purpose. Thus, by cleanly shutting down the old-style OSD process, leaving just the container-based OSD running, draining that OSD, wiping out everything that automated cleanup missed and re-creating the OSD, all of the data in the OSD is going to have passed through the migration process twice, and I would expect migration to detect and clean up (or at least report) any inconsistencies, so that they don't pop up months or years later. Granted, if you have triple redundancy on the pools in the OSD, the likelihood of lost/damaged data is pretty low, but in my case, I was still recovering from a fried Internet connection and didn't want any more surprises. So ultimately the choice is yours, Quick-fix or slow. Tim On Sat, 2024-08-17 at 08:05 +0000, Eugen Block wrote:
Hi,
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish.
note that there are still users/operators/admins who don't use containers. Changing the ceph-volume default might not be the best idea in this case.
Regarding the cleanup, this was the thread [1] Tim was referring to. I would set the noout flag, stop an OSD (so the device won't be busy anymore), make sure that both ceph-osd@{OSD_ID} and ceph-{FSID}@osd.{OSD_ID} then double check that everything you need is still under /var/lib/ceph/{FSID}/osd.{OSD_ID}, like configs an keyrings. Disable the ceph-osd@{OSD_ID} (as already pointed out), then check if the orchestrator can start the OSD via systemd:
ceph orch daemon start osd.{OSD_ID}
or alternatively, try it manually:
systemctl reset-failed systemctl start ceph-{FSID}@osd.{OSD_ID}
Watch the log for that OSD to identify any issues. If it works, unset the noout flag. You might want to ensure it also works after a reboot, though. I don't think it should be necessary to redeploy the OSDs, but the cleanup has to be proper. As a guidance you can check the cephadm tool's contents and look for the "adopt" function. That migrates the contents of the pre-cephadm daemons into the FSID specific directories.
Regards, Eugen
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/K2R3MXRD3S2...
Zitat von Dan O'Brien <dobrie2@gmu.edu>:
OK... I've been in the Circle of Hell where systemd lives and I *THINK* I have convinced myself I'm OK. I *REALLY* don't want to trash and rebuild the OSDs.
In the manpage for systemd.unit, I found
UNIT GARBAGE COLLECTION The system and service manager loads a unit's configuration automatically when a unit is referenced for the first time. It will automatically unload the unit configuration and state again when the unit is not needed anymore ("garbage collection").
I've disabled the systemd units (which removes the symlink from the target) for the non-cephadm OSDs I created by mistake and I'm PRETTY SURE if I wait long enough (or reboot) that I won't see them any more, since there won't be a unit for systemd to care about.
I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just now, but it says "device busy." I think that's because there's some OTHER systemd cruft that shows a mount: [root@ceph02 ~]# systemctl --all | grep ceph | grep mount var-lib-ceph-osd-ceph\x2d11.mount loaded active mounted /var/lib/ceph/osd/ceph-11 var-lib-ceph-osd-ceph\x2d25.mount loaded active mounted /var/lib/ceph/osd/ceph-25 var-lib-ceph-osd-ceph\x2d9.mount loaded active mounted /var/lib/ceph/osd/ceph-9
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Adam Tygart
-
Dan O'Brien
-
Eugen Block
-
Tim Holloway