2 Oct
2020
2 Oct
'20
8:37 a.m.
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a
single new OSD daemon with a db device on SSD. But I can’t find any documentation about
this.
I have tried:
1. Using web dashboard. This requires at least one filter to proceed (type, vendor,
model or size). But I just want to select the block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db
device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But
ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’
does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I
can’t figure out what is that.
Any help is appreciated. Thanks.
2 Oct
2 Oct
12:58 p.m.
Hi,
at the moment there's only the manual way to deploy single OSDs, not
with cephadm. There have been a couple of threads on this list, I
don't have a link though.
You'll have to run something like
ceph-volume lvm create --data /dev/sdX --block.db {VG/LV}
Note that for block.db you'll need to provide the volume-goup/logical
volume, not the device path.
Regards,
Eugen
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm.
Now I want to add a single new OSD daemon with a db device on SSD.
But I can’t find any documentation about this.
I have tried:
1. Using web dashboard. This requires at least one filter to
proceed (type, vendor, model or size). But I just want to select the
block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot
specify my SSD db device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume
prepare and activate. But ceph-volume seems can’t create systemd
service outside the container like ‘ceph orch’ does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires
a json config file, I can’t figure out what is that.
Any help is appreciated. Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:43 p.m.
Thanks. You mean directly running ‘ceph-volume lvm create’ on target host (not inside any
container like what ‘ceph orch’ does), right?
And I finally found a hack way to run my OSD in a container.
1. ceph orch daemon add osd host:/dev/sdX
2. On target host, stop the just created OSD service.
3. ‘ceph osd destroy’ the just created osd.
4. On target host, run ‘cephadm shell’
* ceph-volume lvm zap —destroy /dev/sdX
* ceph-volume lvm prepare —data /dev/sdX —block.db vg/lv —osd-id x —osd-fsid xxxx
—no-systemd This replaced the auto created OSD with my desired config and reuse the
previous ID and fsid.
5. On target host, restart the OSD service.
I think the OSD created in this way fits better into other ‘ceph orch’ operations. Any
advice on this?
On Oct 2, 2020, at 20:59, Eugen Block <eblock(a)nde.ag> wrote:
Hi,
at the moment there's only the manual way to deploy single OSDs, not with cephadm.
There have been a couple of threads on this list, I don't have a link though.
You'll have to run something like
ceph-volume lvm create --data /dev/sdX --block.db {VG/LV}
Note that for block.db you'll need to provide the volume-goup/logical volume, not the
device path.
Regards,
Eugen
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a
single new OSD daemon with a db device on SSD. But I can’t find any documentation about
this.
I have tried:
1. Using web dashboard. This requires at least one filter to proceed (type, vendor,
model or size). But I just want to select the block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device
in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But
ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’
does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I
can’t figure out what is that.
Any help is appreciated. Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:18 p.m.
Doing it in the container seems the right way and you also seem to get
it running. I didn’t have the time to dig into cephadm yet, so my
knowledge is too limited at this point. But I think you could skip the
creation and wiping and just run the create command within the
container.
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Thanks. You mean directly running ‘ceph-volume lvm
create’ on target
host (not inside any container like what ‘ceph orch’ does), right?
And I finally found a hack way to run my OSD in a container.
1. ceph orch daemon add osd host:/dev/sdX
2. On target host, stop the just created OSD service.
3. ‘ceph osd destroy’ the just created osd.
4. On target host, run ‘cephadm shell’
* ceph-volume lvm zap —destroy /dev/sdX
* ceph-volume lvm prepare —data /dev/sdX —block.db vg/lv
—osd-id x —osd-fsid xxxx —no-systemd This replaced the auto created
OSD with my desired config and reuse the previous ID and fsid.
5. On target host, restart the OSD service.
I think the OSD created in this way fits better into other ‘ceph
orch’ operations. Any advice on this?
On Oct 2, 2020, at 20:59, Eugen Block <eblock(a)nde.ag> wrote:
Hi,
at the moment there's only the manual way to deploy single OSDs, not
with cephadm. There have been a couple of threads on this list, I
don't have a link though.
You'll have to run something like
ceph-volume lvm create --data /dev/sdX --block.db {VG/LV}
Note that for block.db you'll need to provide the
volume-goup/logical volume, not the device path.
Regards,
Eugen
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm.
Now I want to add a single new OSD daemon with a db device on SSD.
But I can’t find any documentation about this.
I have tried:
1. Using web dashboard. This requires at least one filter to
proceed (type, vendor, model or size). But I just want to select the
block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot
specify my SSD db device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume
prepare and activate. But ceph-volume seems can’t create systemd
service outside the container like ‘ceph orch’ does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires
a json config file, I can’t figure out what is that.
Any help is appreciated. Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3 Dec
3 Dec
7:05 a.m.
I finally found out how to create a single OSD manually with ceph-volume, cephadm, but
without create, destroy and recreate the OSD. The point is that ceph-volume does not
understand container. It will create the systemd unit in the container which will not
work. I need to use cephadm to create the systemd unit outside the container, which will
invoke docker/podman.
To help other people, here is the step-by-step instructions:
1. Copy the output “ceph config generate-minimal-conf” to /etc/ceph/ceph.conf on the
host you want to deploy new OSDs.
2. Run “cephadm shell -m /var/lib/ceph” on OSD host. This will mount /var/lib/ceph on
the host to /mnt/ceph in the container. Now you are in a shell in the container.
3. Copy the output of “ceph auth export client.bootstrap-osd” to
/var/lib/ceph/bootstrap-osd/ceph.keyring in the container.
4. Run “ceph-volume prepare --no-systemd” and add any arguments you want, like --data,
--block.db
5. Run “cp -r /var/lib/ceph/osd /mnt/ceph/”. This will preserve files created by
ceph-volume after the container terminates.
6. Exit the shell in the container.
7. Run “cephadm --image ceph/ceph:v15.2.6 adopt --style legacy --name osd.X” (replace
with your ceph version and osd ID)
8. Finally, Run “systemctl start ceph-<fsid>(a)osd.X.xn--service-b46c to start the OSD
daemon.
Thank you for all your helps. Although without destroying and recreating, this procedure
is still too complicated. I understand that manually deploy a single OSD is not very
common, especially in large scale deployment. But our deployment is tiny. We reuse
existing servers, and every host has different disk layout. If “ceph orch daemon add osd
host:/dev/sdX” command allow us to pass arbitrary additional arguments to “ceph-volume”,
it would be a lot more convenient.
发件人: Eugen Block<mailto:eblock@nde.ag>
发送时间: 2020年10月3日 1:18
收件人: 胡 玮文<mailto:huww98@outlook.com>
抄送: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
主题: Re: How to create single OSD with SSD db device with cephadm
Doing it in the container seems the right way and you also seem to get
it running. I didn’t have the time to dig into cephadm yet, so my
knowledge is too limited at this point. But I think you could skip the
creation and wiping and just run the create command within the
container.
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Thanks. You mean directly running ‘ceph-volume lvm
create’ on target
host (not inside any container like what ‘ceph orch’ does), right?
And I finally found a hack way to run my OSD in a container.
1. ceph orch daemon add osd host:/dev/sdX
2. On target host, stop the just created OSD service.
3. ‘ceph osd destroy’ the just created osd.
4. On target host, run ‘cephadm shell’
* ceph-volume lvm zap —destroy /dev/sdX
* ceph-volume lvm prepare —data /dev/sdX —block.db vg/lv
—osd-id x —osd-fsid xxxx —no-systemd This replaced the auto created
OSD with my desired config and reuse the previous ID and fsid.
5. On target host, restart the OSD service.
I think the OSD created in this way fits better into other ‘ceph
orch’ operations. Any advice on this?
On Oct 2, 2020, at 20:59, Eugen Block <eblock(a)nde.ag> wrote:
Hi,
at the moment there's only the manual way to deploy single OSDs, not
with cephadm. There have been a couple of threads on this list, I
don't have a link though.
You'll have to run something like
ceph-volume lvm create --data /dev/sdX --block.db {VG/LV}
Note that for block.db you'll need to provide the
volume-goup/logical volume, not the device path.
Regards,
Eugen
Zitat von 胡 玮文 <huww98(a)outlook.com>om>:
Hi all,
I’m new to ceph. I recently deployed a ceph cluster with cephadm.
Now I want to add a single new OSD daemon with a db device on SSD.
But I can’t find any documentation about this.
I have tried:
1. Using web dashboard. This requires at least one filter to
proceed (type, vendor, model or size). But I just want to select the
block device manually.
2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based.
3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot
specify my SSD db device in this way.
4. On the target host, run ‘cephadm shell’ then ceph-volume
prepare and activate. But ceph-volume seems can’t create systemd
service outside the container like ‘ceph orch’ does.
5. On the target host, run ‘cephadm ceph-volume’, but it requires
a json config file, I can’t figure out what is that.
Any help is appreciated. Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1252
days inactive
1314
days old
4 comments
2 participants
participants (2)
-
Eugen Block -
胡 玮文