Hi Tim, On Mon, 15 Jul 2024 at 07:51, Eugen Block <eblock@nde.ag> wrote:
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status.
Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -> '/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
By any chance that ceph was accidentally installed on those systems? As it sounds very much like systemd is activating the OSDs through ceph-volume.
Another maybe related issue if seen while upgrading (some time ago), the FSID changed somehow and I ended up with OSDs that had the wrong FSID. Cheers, Alwin