Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree". --- More seriously, I've been having problems with OSDs that report as being both up and down at the same time. This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM. But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz: ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy Note that truncated message above reads bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why. --- Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN". Here's what systemctl thinks about it: systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory The actual container is not running. Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly).
Hi, containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5 Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about. Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This particular system has it both ways and neither wants to work. The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try. ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504 Here's the OSD tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000 Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine. And here's the host list as the orchestrator sees it: ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system- ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example: ceph osd crush remove www2 ceph osd crush remove docker0 and so on. Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one: systemctl disable --now ceph-osd@4 and see if the container takes over. Was this your attempt to adopt an existing OSD from pre-cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device: cephadm [--image your-custom-image] adopt --style legacy --name osd.4 locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway. [1] https://docs.ceph.com/en/latest/cephadm/adoption/ Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system- ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it. On the legacy/container OSD stuff, that is a complete puzzle. While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems. The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot. This is what happens when I try to adopt: cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -> '/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block' I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now. On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example:
ceph osd crush remove www2 ceph osd crush remove docker0
and so on.
Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one:
systemctl disable --now ceph-osd@4
and see if the container takes over.
Was this your attempt to adopt an existing OSD from pre-cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device:
cephadm [--image your-custom-image] adopt --style legacy --name osd.4
locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway.
[1] https://docs.ceph.com/en/latest/cephadm/adoption/
Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system- ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status. Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -> '/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example:
ceph osd crush remove www2 ceph osd crush remove docker0
and so on.
Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one:
systemctl disable --now ceph-osd@4
and see if the container takes over.
Was this your attempt to adopt an existing OSD from pre-cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device:
cephadm [--image your-custom-image] adopt --style legacy --name osd.4
locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway.
[1] https://docs.ceph.com/en/latest/cephadm/adoption/
Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system- ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, On Mon, 15 Jul 2024 at 07:51, Eugen Block <eblock@nde.ag> wrote:
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status.
Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -> '/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
By any chance that ceph was accidentally installed on those systems? As it sounds very much like systemd is activating the OSDs through ceph-volume.
Another maybe related issue if seen while upgrading (some time ago), the FSID changed somehow and I ended up with OSDs that had the wrong FSID. Cheers, Alwin
The problem with merely disabling or masking the non-cephadm OSD is that the offending systemd service unit lives under /run, not under /lib/systemd or /etc/systemd. As far as I know, essentially the entire /run directory's contents get destroyed when you reboot and that would include the disabled OSD unit. Then a new copy would get created as the system boot proceeded. I could, of course, then re-disable it, but that's not a very pretty solution. Better to determine why Ceph feels the need to create this Systemd service dynamically and persuade it not to. I was kind of hoping that it came from finding that OSD directory that under /var/lib/ceph/osd, but as I said, I have another machine with TWO such directories and only one manifests as a systemd service. The other doesn't run at all, doesn't list on an osd tree, orch ps or dashboard, and since as far as I'm concerned doesn't exist anyway, I'll not complain about that. I just need to get the invalid stuff excised safely. Oh wait, one difference between the two /var/lib/cep/osd's is that the one that's running has files, the one that isn't is just an empty directory. Which suggests that the cue for making the /run/ceph/osd service may be the detection of one of the files there and maybe I could risk ripping the unwanted directory out. I thinks there are some softlinks, though, so I'll proceed with caution. On the plus side, the autotuner seems to have finally kicked in. First time I've seen "HEALTH OK" in a while! Alwin: Thanks for your interest. Both the funny machines are dedicated ceph host nodes, so it's no accident that cephadm is installed on the. And I've never had more than 1 fsid, so no issue there. If you're thinking about the "phantom host", that was just because of a typing error when adding a new ceph host. That problem has now been resolved. Tim On Mon, 2024-07-15 at 05:49 +0000, Eugen Block wrote:
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status.
Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -
'/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example:
ceph osd crush remove www2 ceph osd crush remove docker0
and so on.
Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one:
systemctl disable --now ceph-osd@4
and see if the container takes over.
Was this your attempt to adopt an existing OSD from pre-cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device:
cephadm [--image your-custom-image] adopt --style legacy --name osd.4
locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway.
[1] https://docs.ceph.com/en/latest/cephadm/adoption/
Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI- AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree".
---
More seriously, I've been having problems with OSDs that report as being both up and down at the same time.
This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM.
But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz:
ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86- 0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- 12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph- 4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- 278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- 12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- 278fcd86-0861- 11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- 12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy
Note that truncated message above reads
bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy
Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why.
---
Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN".
Here's what systemctl thinks about it:
systemctl status ceph-osd@5.service ● ceph-osd@5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph- osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system- ceph\x2dosd.slice/ceph-osd@5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph -- id 5 -- setuser ceph --setgroup ceph
Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024- 07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024- 07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024- 07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
The actual container is not running.
Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do you have more ceph packages installed than just cephadm? If you have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove them and clean up the directories properly. To me it looks like a mixup of "traditional" package based installation and cephadm deployment. Only you can tell how and why, but it's more important to clean that up and keep it consistent. You should keep the cephadm and optional the ceph-common package, but the rest isn't required to run a cephadm cluster. Zitat von Tim Holloway <timh@mousetech.com>:
The problem with merely disabling or masking the non-cephadm OSD is that the offending systemd service unit lives under /run, not under /lib/systemd or /etc/systemd.
As far as I know, essentially the entire /run directory's contents get destroyed when you reboot and that would include the disabled OSD unit. Then a new copy would get created as the system boot proceeded. I could, of course, then re-disable it, but that's not a very pretty solution. Better to determine why Ceph feels the need to create this Systemd service dynamically and persuade it not to.
I was kind of hoping that it came from finding that OSD directory that under /var/lib/ceph/osd, but as I said, I have another machine with TWO such directories and only one manifests as a systemd service. The other doesn't run at all, doesn't list on an osd tree, orch ps or dashboard, and since as far as I'm concerned doesn't exist anyway, I'll not complain about that. I just need to get the invalid stuff excised safely.
Oh wait, one difference between the two /var/lib/cep/osd's is that the one that's running has files, the one that isn't is just an empty directory. Which suggests that the cue for making the /run/ceph/osd service may be the detection of one of the files there and maybe I could risk ripping the unwanted directory out. I thinks there are some softlinks, though, so I'll proceed with caution.
On the plus side, the autotuner seems to have finally kicked in. First time I've seen "HEALTH OK" in a while!
Alwin: Thanks for your interest. Both the funny machines are dedicated ceph host nodes, so it's no accident that cephadm is installed on the. And I've never had more than 1 fsid, so no issue there.
If you're thinking about the "phantom host", that was just because of a typing error when adding a new ceph host. That problem has now been resolved.
Tim
On Mon, 2024-07-15 at 05:49 +0000, Eugen Block wrote:
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status.
Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -
'/var/lib/ceph/278fcd86-0861-11ee-a7df-9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example:
ceph osd crush remove www2 ceph osd crush remove docker0
and so on.
Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one:
systemctl disable --now ceph-osd@4
and see if the container takes over.
Was this your attempt to adopt an existing OSD from pre-cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device:
cephadm [--image your-custom-image] adopt --style legacy --name osd.4
locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway.
[1] https://docs.ceph.com/en/latest/cephadm/adoption/
Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI- AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote:
Hi,
containerized daemons usually have the fsid in the systemd unit, like ceph-{fsid}@osd.5
Is it possible that you have those confused? Check the /var/lib/ceph/osd/ directory to find possible orphaned daemons and clean them up. And as previously stated, it would help to see your osd tree and which OSDs you’re talking about.
Zitat von Tim Holloway <timh@mousetech.com>:
> Incidentally, I just noticed that my phantom host isn't > completely > gone. It's not in the host list, either command-line or > dashboard, > but > it does list (with no assets) as a host under "ceph osd > tree". > > --- > > More seriously, I've been having problems with OSDs that > report as > being both up and down at the same time. > > This is on 2 new hosts. One host saw this when I made it > the _admin > host. The other caught it because it's running in a VM with > the OSD > mapped out as an imported disk and the host os managed to > flip > which > drive was sda and which was sbd, resulting in having to > delete and > re- > define the OSD in the VM. > > But now the OSD on this VM reports as "UP/IN" on the > dashboard > while > it's "error" on "ceph orch ps" and on the actual vbox, the > osd > container fails on startup. viz: > > ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86- > 0861- > 11ee- > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > 12T20:06:48.056+0000 > 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph- > 4/block) > open open got: (16) De> > Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- > 278fcd86-0861- > 11ee- > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > 12T20:06:48.056+0000 > 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object > store > Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- > 278fcd86-0861- > 11ee- > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > 12T20:06:48.056+0000 > 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or > resource > busy > > Note that truncated message above reads > > bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open > open got: > (16) > Device or resource busy > > Rebooting doesn't help. Nor does freeing us resources and > stopping/starting processes manually. The problem > eventually > cleared up > spontaneously on the admin box, but I have no idea why. > > --- > > Also noted that now the OSD on the admin box shows in ceph > orch ps > as > "stopped", though again, the dashboard lists it as "UP/IN". > > Here's what systemctl thinks about it: > > systemctl status ceph-osd@5.service > ● ceph-osd@5.service - Ceph object storage daemon osd.5 > Loaded: loaded (/usr/lib/systemd/system/ceph- > osd@.service; > enabled-runtime; preset: disabled) > Active: active (running) since Fri 2024-07-12 16:45:51 > EDT; > 1min > 40s ago > Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd- > prestart.sh - > -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) > Main PID: 8517 (ceph-osd) > Tasks: 70 > Memory: 478.6M > CPU: 3.405s > CGroup: /system.slice/system- > ceph\x2dosd.slice/ceph-osd@5.service > └─8517 /usr/bin/ceph-osd -f --cluster ceph -- > id 5 -- > setuser ceph --setgroup ceph > > Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting > Ceph > object > storage daemon osd.5... > Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started > Ceph > object > storage daemon osd.5. > Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024- > 07- > 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public > interface > Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024- > 07- > 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 > log_to_monitors > {default=true} > Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024- > 07- > 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 > set_numa_affinity > unable to identify public interface '' numa node: (2) No > such file > or > directory > > The actual container is not running. > > Ceph version, incidentally, is 16.2.15. Except for that one > node > that > apparently didn't move up from Octupus (I'll be nuking that > one > shortly). > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting. And thanks for the info. I did a quick look-around. The admin node, which is one of the mixed- osd machines has these packages installed: centos-release-ceph-pacific-1.0-2.el9.noarch cephadm-16.2.14-2.el9s.noarch libcephfs2-16.2.15-1.el9s.x86_64 python3-ceph-common-16.2.15-1.el9s.x86_64 python3-ceph-argparse-16.2.15-1.el9s.x86_64 python3-cephfs-16.2.15-1.el9s.x86_64 libcephsqlite-16.2.15-1.el9s.x86_64 ceph-common-16.2.15-1.el9s.x86_64 ceph-base-16.2.15-1.el9s.x86_64 ceph-selinux-16.2.15-1.el9s.x86_64 ceph-mds-16.2.15-1.el9s.x86_64 ceph-mon-16.2.15-1.el9s.x86_64 ceph-osd-16.2.15-1.el9s.x86_64 ceph-prometheus-alerts-16.2.15-1.el9s.noarch ceph-grafana-dashboards-16.2.15-1.el9s.noarch ceph-mgr-dashboard-16.2.15-1.el9s.noarch ceph-mgr-diskprediction-local-16.2.15-1.el9s.noarch ceph-mgr-k8sevents-16.2.15-1.el9s.noarch ceph-mgr-modules-core-16.2.15-1.el9s.noarch ceph-mgr-16.2.15-1.el9s.x86_64 ceph-mgr-rook-16.2.15-1.el9s.noarch ceph-16.2.15-1.el9s.x86_64 The other problem node has these: centos-release-ceph-pacific-1.0-2.el9.noarch python3-ceph-common-16.2.15-1.el9s.x86_64 libcephfs2-16.2.15-1.el9s.x86_64 python3-ceph-argparse-16.2.15-1.el9s.x86_64 python3-cephfs-16.2.15-1.el9s.x86_64 libcephsqlite-16.2.15-1.el9s.x86_64 ceph-mgr-modules-core-16.2.15-1.el9s.noarch ceph-common-16.2.15-1.el9s.x86_64 ceph-base-16.2.15-1.el9s.x86_64 ceph-selinux-16.2.15-1.el9s.x86_64 ceph-mgr-diskprediction-local-16.2.15-1.el9s.noarch ceph-mgr-16.2.15-1.el9s.x86_64 ceph-mds-16.2.15-1.el9s.x86_64 ceph-mon-16.2.15-1.el9s.x86_64 ceph-osd-16.2.15-1.el9s.x86_64 ceph-16.2.15-1.el9s.x86_64 cephadm-16.2.14-2.el9s.noarch But a simple node hosting only and OSD and NFS did look like this: centos-release-ceph-pacific-1.0-2.el9.noarch cephadm-16.2.14-2.el9s.noarch I did an in-place migration from AlmaLinux 8 to AlamaLinux 9 and that may have had side-effects. But who knows? I'll start ripping packages and see what happens. On Tue, 2024-07-16 at 06:38 +0000, Eugen Block wrote:
Do you have more ceph packages installed than just cephadm? If you have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove them and clean up the directories properly. To me it looks like a mixup of "traditional" package based installation and cephadm deployment. Only you can tell how and why, but it's more important to clean that up and keep it consistent. You should keep the cephadm and optional the ceph-common package, but the rest isn't required to run a cephadm cluster.
Zitat von Tim Holloway <timh@mousetech.com>:
The problem with merely disabling or masking the non-cephadm OSD is that the offending systemd service unit lives under /run, not under /lib/systemd or /etc/systemd.
As far as I know, essentially the entire /run directory's contents get destroyed when you reboot and that would include the disabled OSD unit. Then a new copy would get created as the system boot proceeded. I could, of course, then re-disable it, but that's not a very pretty solution. Better to determine why Ceph feels the need to create this Systemd service dynamically and persuade it not to.
I was kind of hoping that it came from finding that OSD directory that under /var/lib/ceph/osd, but as I said, I have another machine with TWO such directories and only one manifests as a systemd service. The other doesn't run at all, doesn't list on an osd tree, orch ps or dashboard, and since as far as I'm concerned doesn't exist anyway, I'll not complain about that. I just need to get the invalid stuff excised safely.
Oh wait, one difference between the two /var/lib/cep/osd's is that the one that's running has files, the one that isn't is just an empty directory. Which suggests that the cue for making the /run/ceph/osd service may be the detection of one of the files there and maybe I could risk ripping the unwanted directory out. I thinks there are some softlinks, though, so I'll proceed with caution.
On the plus side, the autotuner seems to have finally kicked in. First time I've seen "HEALTH OK" in a while!
Alwin: Thanks for your interest. Both the funny machines are dedicated ceph host nodes, so it's no accident that cephadm is installed on the. And I've never had more than 1 fsid, so no issue there.
If you're thinking about the "phantom host", that was just because of a typing error when adding a new ceph host. That problem has now been resolved.
Tim
On Mon, 2024-07-15 at 05:49 +0000, Eugen Block wrote:
If the OSD is already running in a container, adopting it won't work, as you already noticed. I don't have an explanation how the non-cephadm systemd unit has been created, but that should be fixed by disabling it.
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
Check which OSDs are active and remove the remainders of the orphaned directories, that should be fine. But be careful and check properly before actually remocing anything and only remove one by one while watching the cluster status.
Zitat von Tim Holloway <timh@mousetech.com>:
OK. Phantom hosts are gone. Many thanks! I'll have to review my checklist for decomissioning hosts to make sure that step is on it.
On the legacy/container OSD stuff, that is a complete puzzle.
While the first thing that I see when I look up "creating an OSD" in the system documentation is the manual process, I've been using cephadm long enough to know to dig past that. The manual process is sufficiently tedious that I cannot think that I'd have used it by accident. Especially since I set out with the explicit goal of using cephadm. Yet here it is. This isn't an upgraded machine, it was constructed within the last week from the ground up. So I have no idea how the legacy definition got there. On two separate systems.
The disable on the legacy OSD worked and the container is now running. Although I'm not sure that it will survive a reboot, since the legacy service is dynamically created on each reboot.
This is what happens when I try to adopt:
cephadm adopt --style legacy --name osd.4 Pulling container image quay.io/ceph/ceph:v16... Found online OSD at //var/lib/ceph/osd/ceph-4/fsid objectstore_type is bluestore Disabling old systemd unit ceph-osd@4... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9509, in <module> main() File "/usr/sbin/cephadm", line 9497, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2061, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6043, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6210, in command_adopt_ceph move_files(ctx, glob(os.path.join(data_dir_src, '*')), File "/usr/sbin/cephadm", line 2215, in move_files os.symlink(src_rl, dst_file) FileExistsError: [Errno 17] File exists: '/dev/vg_ceph/ceph0504' -
'/var/lib/ceph/278fcd86-0861-11ee-a7df- 9c5c8e86cf8f/osd.4/block'
I have considered simply doing a brute-force removal of the OSD files in /var/lib/ceph/osd but I'm not sure what ill effects might ensue. I discovered that my other offending machine actually has TWO legacy OSD directories, but only one of them is being used. The other OSD is the remnant of a deletion and it's just dead files now.
On 7/13/24 02:39, Eugen Block wrote:
Okay, it looks like you just need some further cleanup regarding your phantom hosts, for example:
ceph osd crush remove www2 ceph osd crush remove docker0
and so on.
Regarding the systemd unit (well, cephadm also generates one, but with the fsid as already mentioned), you could just stop and disable the old one:
systemctl disable --now ceph-osd@4
and see if the container takes over.
Was this your attempt to adopt an existing OSD from pre- cephadm?
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
The recommended way would have been to adopt the device:
cephadm [--image your-custom-image] adopt --style legacy -- name osd.4
locally on that host. The --image parameter is optional. Did you follow the docs [1] when you moved to cephadm? Anyway, since it somehow seems to work already, it's probably not that relevant anymore, I just wanted to point to it anyway.
[1] https://docs.ceph.com/en/latest/cephadm/adoption/
Zitat von Tim Holloway <timh@mousetech.com>:
This particular system has it both ways and neither wants to work.
The peculiar thing was that when I first re-created the OSD with cephadm, it was reported that this was an "unmanaged node". So I ran the same cephadm agin and THAT time it showed up. So I suspect that the ceph-osd@4.service was the first install and the ceph-~~~~~@osd.4.servuce got added on the second try.
ceph orch daemon add osd ceph05.internal.mousetech.com:vg_ceph/ceph0504
Here's the OSD tree:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI- AFF -1 7.00000 root default -25 1.00000 host ceph01 1 hdd 1.00000 osd.1 up 1.00000 1.00000 -28 1.00000 host ceph03 3 hdd 1.00000 osd.3 up 1.00000 1.00000 -11 1.00000 host ceph04 7 hdd 1.00000 osd.7 up 1.00000 1.00000 -31 1.00000 host ceph05 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -9 1.00000 host ceph06 2 hdd 1.00000 osd.2 up 1.00000 1.00000 -22 1.00000 host dell02 5 ssd 1.00000 osd.5 up 1.00000 1.00000 -13 0 host docker0 -5 0 host www2 -3 0 host www6 -7 1.00000 host www7 0 hdd 1.00000 osd.0 up 1.00000 1.00000
Note that www2 was recommissioned and replaced by the ceph05 machine and no longer physically exists. www6 was the original/admin ceph node with all the accumulated glop. It ran on the base OS (not in a VM) and I have not attempted to re-create that. Ceph remembers it from its own internal memory, as the original www6 system drive was replaced with an SSD and the OS was installed from scratch. The docker0 machine was never a ceph node. It's the phantom that got yanked it because I accidentally supplied its IP address when doing a ceph orch host add for a completely different machine.
And here's the host list as the orchestrator sees it:
ceph orch host ls HOST ADDR LABELS STATUS ceph01.internal.mousetech.com 10.0.1.21 ceph03.internal.mousetech.com 10.0.1.53 ceph04.mousetech.com 10.0.1.14 ceph05.internal.mousetech.com 10.0.1.54 ceph06.internal.mousetech.com 10.0.1.56 dell02.mousetech.com 10.0.1.52 _admin rgw www7.mousetech.com 10.0.1.7 rgw 7 hosts in cluster
On Fri, 2024-07-12 at 22:15 +0000, Eugen Block wrote: > Hi, > > containerized daemons usually have the fsid in the > systemd > unit, > like > ceph-{fsid}@osd.5 > > Is it possible that you have those confused? Check the > /var/lib/ceph/osd/ directory to find possible orphaned > daemons and > clean them up. > And as previously stated, it would help to see your osd > tree > and > which > OSDs you’re talking about. > > Zitat von Tim Holloway <timh@mousetech.com>: > > > Incidentally, I just noticed that my phantom host isn't > > completely > > gone. It's not in the host list, either command-line or > > dashboard, > > but > > it does list (with no assets) as a host under "ceph osd > > tree". > > > > --- > > > > More seriously, I've been having problems with OSDs > > that > > report as > > being both up and down at the same time. > > > > This is on 2 new hosts. One host saw this when I made > > it > > the _admin > > host. The other caught it because it's running in a VM > > with > > the OSD > > mapped out as an imported disk and the host os managed > > to > > flip > > which > > drive was sda and which was sbd, resulting in having to > > delete and > > re- > > define the OSD in the VM. > > > > But now the OSD on this VM reports as "UP/IN" on the > > dashboard > > while > > it's "error" on "ceph orch ps" and on the actual vbox, > > the > > osd > > container fails on startup. viz: > > > > ul 12 20:06:48 ceph05.internal.mousetech.com ceph- > > 278fcd86- > > 0861- > > 11ee- > > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > > 12T20:06:48.056+0000 > > 7fc17dfb9380 -1 bdev(0x55e4853c4800 > > /var/lib/ceph/osd/ceph- > > 4/block) > > open open got: (16) De> > > Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- > > 278fcd86-0861- > > 11ee- > > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > > 12T20:06:48.056+0000 > > 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount > > object > > store > > Jul 12 20:06:48 ceph05.internal.mousetech.com ceph- > > 278fcd86-0861- > > 11ee- > > a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07- > > 12T20:06:48.056+0000 > > 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device > > or > > resource > > busy > > > > Note that truncated message above reads > > > > bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) > > open > > open got: > > (16) > > Device or resource busy > > > > Rebooting doesn't help. Nor does freeing us resources > > and > > stopping/starting processes manually. The problem > > eventually > > cleared up > > spontaneously on the admin box, but I have no idea why. > > > > --- > > > > Also noted that now the OSD on the admin box shows in > > ceph > > orch ps > > as > > "stopped", though again, the dashboard lists it as > > "UP/IN". > > > > Here's what systemctl thinks about it: > > > > systemctl status ceph-osd@5.service > > ● ceph-osd@5.service - Ceph object storage daemon osd.5 > > Loaded: loaded (/usr/lib/systemd/system/ceph- > > osd@.service; > > enabled-runtime; preset: disabled) > > Active: active (running) since Fri 2024-07-12 > > 16:45:51 > > EDT; > > 1min > > 40s ago > > Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph- > > osd- > > prestart.sh - > > -cluster ${CLUSTER} --id 5 (code=exited, > > status=0/SUCCESS) > > Main PID: 8517 (ceph-osd) > > Tasks: 70 > > Memory: 478.6M > > CPU: 3.405s > > CGroup: /system.slice/system- > > ceph\x2dosd.slice/ceph-osd@5.service > > └─8517 /usr/bin/ceph-osd -f --cluster ceph > > -- > > id 5 -- > > setuser ceph --setgroup ceph > > > > Jul 12 16:45:51 dell02.mousetech.com systemd[1]: > > Starting > > Ceph > > object > > storage daemon osd.5... > > Jul 12 16:45:51 dell02.mousetech.com systemd[1]: > > Started > > Ceph > > object > > storage daemon osd.5. > > Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: > > 2024- > > 07- > > 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to > > public > > interface > > Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: > > 2024- > > 07- > > 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 > > log_to_monitors > > {default=true} > > Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: > > 2024- > > 07- > > 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 > > set_numa_affinity > > unable to identify public interface '' numa node: (2) > > No > > such file > > or > > directory > > > > The actual container is not running. > > > > Ceph version, incidentally, is 16.2.15. Except for that > > one > > node > > that > > apparently didn't move up from Octupus (I'll be nuking > > that > > one > > shortly). > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to > > ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
OK. I deleted the questionable stuff with this command: dnf erase ceph-mgr-modules-core-16.2.15-1.el9s.noarch ceph-mgr- diskprediction-local-16.2.15-1.el9s.noarch ceph-mgr-16.2.15- 1.el9s.x86_64 ceph-mds-16.2.15-1.el9s.x86_64 ceph-mon-16.2.15- 1.el9s.x86_64 That left these two: centos-release-ceph-pacific-1.0-2.el9.noarch cephadm-16.2.14-2.el9s.noarch At that point, I couldn't run cephadm because the /etc/ceph directory was deleted as well by dnf. I copied it back in from the master copy but no containers were running. A reboot brought everything back up. So I'm optimistic that that machine is now clean (and has nearly 500MB of newly-freed space from the deleted packages!) One of the deleted items was the "ceph" meta-package. I'm thinking that I installed this and it's what pulled in the others. The final machine is operational and I'm going to leave it, but it does show 1 quirk. Dashboard and osd tree show its OSD as up/running, but "ceph orch ps" shows it as "stopped". My guess is that ceph orch is looking for the container OSD and doesn't notice the legacy OSD. Thanks again for all the help! Tim On Tue, 2024-07-16 at 06:38 +0000, Eugen Block wrote:
Do you have more ceph packages installed than just cephadm? If you have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove them and clean up the directories properly. To me it looks like a mixup of "traditional" package based installation and cephadm deployment. Only you can tell how and why, but it's more important to clean that up and keep it consistent. You should keep the cephadm and optional the ceph-common package, but the rest isn't required to run a cephadm cluster.
(history deleted. See earlier copies)
Hi,
The final machine is operational and I'm going to leave it, but it does show 1 quirk. Dashboard and osd tree show its OSD as up/running, but "ceph orch ps" shows it as "stopped". My guess is that ceph orch is looking for the container OSD and doesn't notice the legacy OSD.
I assume so, yes.
Thanks again for all the help!
You're welcome! Zitat von Tim Holloway <timh@mousetech.com>:
OK. I deleted the questionable stuff with this command:
dnf erase ceph-mgr-modules-core-16.2.15-1.el9s.noarch ceph-mgr- diskprediction-local-16.2.15-1.el9s.noarch ceph-mgr-16.2.15- 1.el9s.x86_64 ceph-mds-16.2.15-1.el9s.x86_64 ceph-mon-16.2.15- 1.el9s.x86_64
That left these two:
centos-release-ceph-pacific-1.0-2.el9.noarch cephadm-16.2.14-2.el9s.noarch
At that point, I couldn't run cephadm because the /etc/ceph directory was deleted as well by dnf. I copied it back in from the master copy but no containers were running. A reboot brought everything back up.
So I'm optimistic that that machine is now clean (and has nearly 500MB of newly-freed space from the deleted packages!)
One of the deleted items was the "ceph" meta-package. I'm thinking that I installed this and it's what pulled in the others.
The final machine is operational and I'm going to leave it, but it does show 1 quirk. Dashboard and osd tree show its OSD as up/running, but "ceph orch ps" shows it as "stopped". My guess is that ceph orch is looking for the container OSD and doesn't notice the legacy OSD.
Thanks again for all the help!
Tim
On Tue, 2024-07-16 at 06:38 +0000, Eugen Block wrote:
Do you have more ceph packages installed than just cephadm? If you have ceph-osd packages (or ceph-mon, ceph-mds etc.), I would remove them and clean up the directories properly. To me it looks like a mixup of "traditional" package based installation and cephadm deployment. Only you can tell how and why, but it's more important to clean that up and keep it consistent. You should keep the cephadm and optional the ceph-common package, but the rest isn't required to run a cephadm cluster.
(history deleted. See earlier copies)
participants (3)
-
Alwin Antreich
-
Eugen Block
-
Tim Holloway