It depends on your available resources, but I really do recommend destroying and re-creating that OSD. If you have to spin up a VM and set up a temporary OSD just to keep the overall system happy, even that is a small price to pay. As I said, you can't unlink/disable the container systemd, because it's created and re-recreated dynamically. The non-container systemd unit is easier to deal with, since it's static but if you're sharing resources like I was, I still think it safer if you migrate everything out of that OSD, totally destroy all traces of both realizations of it, then re-create and migrate back. Ceph can do that, and although it can take a while to do the migration, it's well worth the effort, Ceph is very good at keeping data safe, but I'd not abuse the privilege. Oh yeah, you have multiple OSDs with that problem. Do one at a time and you'll go faster, Doing a mass shovel of data in and out of one OSD takes long enough, but trying to run the process in parallel may well bog down the whole Ceph network. As I recall, as part of my own fumbling approach, I first stopped and masked the osd-style systemd unit. But as I said, different subsystems look in different places to monitor OSDs and I'd get conflicting status reports. So while in theory, you could brute-force remove the systemd unit and the /car/lib/ceph/osd.x directory subtree (preferably while the container is also offline, I think it cleaner and safer overall to simply go back to a blank slate. On Fri, 2024-08-16 at 21:05 +0000, Dan O'Brien wrote:
OK... I've been in the Circle of Hell where systemd lives and I *THINK* I have convinced myself I'm OK. I *REALLY* don't want to trash and rebuild the OSDs.
In the manpage for systemd.unit, I found
UNIT GARBAGE COLLECTION The system and service manager loads a unit's configuration automatically when a unit is referenced for the first time. It will automatically unload the unit configuration and state again when the unit is not needed anymore ("garbage collection").
I've disabled the systemd units (which removes the symlink from the target) for the non-cephadm OSDs I created by mistake and I'm PRETTY SURE if I wait long enough (or reboot) that I won't see them any more, since there won't be a unit for systemd to care about.
I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just now, but it says "device busy." I think that's because there's some OTHER systemd cruft that shows a mount: [root@ceph02 ~]# systemctl --all | grep ceph | grep mount var-lib-ceph-osd-ceph\x2d11.mount loaded active mounted /var/lib/ceph/osd/ceph-11 var-lib-ceph-osd-ceph\x2d25.mount loaded active mounted /var/lib/ceph/osd/ceph-25 var-lib-ceph-osd-ceph\x2d9.mount loaded active mounted /var/lib/ceph/osd/ceph-9
When things settle down, I *MIGHT* put in a RFE to change the default for ceph-volume to --no-systemd to save someone else from this anguish. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io