Oh, apparently the last messages weren’t in reply to the list. It seems like removing packages did the trick, and there are up OSDs. Zitat von Eugen Block <eblock@nde.ag>:

Yeah, looks like it worked. :-)

The IDs are part of the crush tree, you don’t need to worry about them. I assume that you could also revert the unmanaged step (Ceph orch set-managed …) and the rest of the OSDs would probably be created as well.

Zitat von Jacek Rużyczka <jacek.ruzyczka@googlemail.com>:

...

...

I would aim at keeping all nodes in the same state, so yes, remove them from all Ceph nodes.

Yep.

Maybe before you reboot, zap the drives first, the

...

ones the orchestrator might have tried to build and failed.

Already done that.

ceph orch set-unmanaged osd.<your_osd_service>

Do you mean this:

root@blade3n1:/# ceph orch set-unmanaged osd.all-available-devices Set unmanaged to True for service osd.all-available-devices

To deploy a single OSD you can run:

...

ceph orch daemon add osd <host> /dev/sdX

So something like this:

mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n1:/dev/nvme0n1 Created osd(s) 3 on host 'blade3n1'

But:

mixtile@blade3n1:~$ sudo ceph orch daemon add osd blade3n2:/dev/nvme0n1 Created no osd(s) on host blade3n2; already created?

Before I could disable the OSD process, it must have started creating OSDs on the other hosts:

mixtile@blade3n1:~$ sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 29.80798 root default -9 7.45200 host blade3n1 3 ssd 7.45200 osd.3 up 1.00000 1.00000 -7 7.45200 host blade3n2 2 ssd 7.45200 osd.2 up 1.00000 1.00000 -5 7.45200 host blade3n3 1 ssd 7.45200 osd.1 up 1.00000 1.00000 -3 7.45200 host blade3n4 0 ssd 7.45200 osd.0 up 1.00000 1.00000

But: What do the negative IDs mean? Yesterday, I only had the root default with ID -1. At least, the health check now says, HEALTH_OK:

mixtile@blade3n1:~$ sudo ceph -s cluster: id: 8aad3073-39a1-11f1-bf6e-f2704a1efa9b health: HEALTH_OK

services: mon: 4 daemons, quorum blade3n1,blade3n2,blade3n3,blade3n4 (age 3h) mgr: blade3n2.cnqdje(active, since 3h), standbys: blade3n1.rrlwwv mds: 1/1 daemons up, 1 standby osd: 4 osds: 4 up (since 10m), 4 in (since 60m)

data: volumes: 1/1 healthy pools: 3 pools, 145 pgs objects: 24 objects, 579 KiB usage: 248 MiB used, 30 TiB / 30 TiB avail pgs: 145 active+clean

So: Do I now have a working Ceph cluster?