14 Jul
2020
14 Jul
'20
1:40 a.m.
Hi,
I have a question regarding support for multiple BLK-MQ queues for Ceph's
RADOS Block Device (RBD). The below given link says that the driver has
been using the BLK-MQ interface for a while but not actually multiple
queues until now with having a queue per-CPU. A change to not hold onto
caps that aren't actually needed. These improvements and more can be found
as part of the Ceph changes for Linux 5.7, which should be released as
stable in early June.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Perf…
My question is: Is it possible that through Ceph FS (Filesystem in User
Space) I can develop a multi-queue driver for Ceph? Asking because this way
I can avoid kernel space. (
https://docs.ceph.com/docs/nautilus/start/quick-cephfs/)
Looking forward for some help
BR
Bobby
15 Jul
15 Jul
4:16 a.m.
New subject: [Ceph-qa] multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD) and CephFS
On Mon, Jul 13, 2020 at 8:40 PM Bobby <italienisch1987(a)gmail.com> wrote:
Hi,
I have a question regarding support for multiple BLK-MQ queues for Ceph's RADOS Block
Device (RBD). The below given link says that the driver has been using the BLK-MQ
interface for a while but not actually multiple queues until now with having a queue
per-CPU. A change to not hold onto caps that aren't actually needed. These
improvements and more can be found as part of the Ceph changes for Linux 5.7, which should
be released as stable in early June.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Perf…
My question is: Is it possible that through Ceph FS (Filesystem in User Space) I can
develop a multi-queue driver for Ceph? Asking because this way I can avoid kernel space.
(https://docs.ceph.com/docs/nautilus/start/quick-cephfs/)
[ trimming CCs to dev and ceph-devel ]
Hi Bobby,
I'm not sure what you mean by a "multi-queue driver for Ceph".
blk-mq is the block layer framework, it has nothing to do with
filesystems, whether local sitting on top of a block device (such
as ext4 or XFS) or distributed sitting on top of a network stack
(such as CephFS).
Do you have a specific project in mind or are you just looking to
make ceph-fuse faster?
Thanks,
Ilya
5:46 a.m.
New subject: [Ceph-qa] multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD) and CephFS
Hi Ilya,
Thanks for the reply. It's basically both i.e. I have a specific project
currently and also I am looking to make ceph-fuse faster.
But for now, let me ask specifically the project based question. In the
project I have to write a blk-mq kernel driver for the Ceph client machine.
The Ceph client machine will transfer the data to HBA or lets say any
embedded device.
My hope is that there can be an alternative and that alternative is to not
implement a blk-mq kernel driver and instead do the stuff in userspace. I
am trying to avoid writing a blk-mq kernel driver and yet achieve the
multi-queue implementation through userspace. Is it possible?
Also AFAIK, the Ceph’s block storage implementation uses a client module
and this client module has two implementations librbd (user-space) and krbd
(kernel module). I have not gone deep into these client modules. but can
librbd help me with this?
Thanks. Looking forward to your reply :-)
BR
On Tue, Jul 14, 2020 at 11:16 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Mon, Jul 13, 2020 at 8:40 PM Bobby
<italienisch1987(a)gmail.com> wrote:
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Perf…
Hi,
I have a question regarding support for multiple BLK-MQ queues for
Ceph's
RADOS Block Device (RBD). The below given link says that the driver
has been using the BLK-MQ interface for a while but not actually multiple
queues until now with having a queue per-CPU. A change to not hold onto
caps that aren't actually needed. These improvements and more can be found
as part of the Ceph changes for Linux 5.7, which should be released as
stable in early June.
My question is: Is it possible that through Ceph FS (Filesystem in User
Space) I
can develop a multi-queue driver for Ceph? Asking because this way
I can avoid kernel space. (
https://docs.ceph.com/docs/nautilus/start/quick-cephfs/)
[ trimming CCs to dev and ceph-devel ]
Hi Bobby,
I'm not sure what you mean by a "multi-queue driver for Ceph".
blk-mq is the block layer framework, it has nothing to do with
filesystems, whether local sitting on top of a block device (such
as ext4 or XFS) or distributed sitting on top of a network stack
(such as CephFS).
Do you have a specific project in mind or are you just looking to
make ceph-fuse faster?
Thanks,
Ilya
16 Jul
16 Jul
2:04 a.m.
New subject: [Ceph-qa] multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD) and CephFS
On Wed, Jul 15, 2020 at 12:47 AM Bobby <italienisch1987(a)gmail.com> wrote:
Hi Ilya,
Thanks for the reply. It's basically both i.e. I have a specific project currently
and also I am looking to make ceph-fuse faster.
But for now, let me ask specifically the project based question. In the project I have to
write a blk-mq kernel driver for the Ceph client machine. The Ceph client machine will
transfer the data to HBA or lets say any embedded device.
What is a "Ceph client machine"?
A Ceph client (or more specifically a RADOS client) speaks RADOS
protocol and transfers data to OSD daemons. It can't transfer data
directly to a physical device because something has to take care of
replication, ensure consistency and self healing, etc. This is the
job of the OSD.
My hope is that there can be an alternative and that alternative is to not implement a
blk-mq kernel driver and instead do the stuff in userspace. I am trying to avoid writing a
blk-mq kernel driver and yet achieve the multi-queue implementation through userspace. Is
it possible?
Also AFAIK, the Ceph’s block storage implementation uses a client module and this client
module has two implementations librbd (user-space) and krbd (kernel module). I have not
gone deep into these client modules. but can librbd help me with this?
I guess I don't understand the goal of your project. A multi-queue
implementation of what exactly? A Ceph block device, a Ceph filesystem
or something else entirely? It would help if you were more specific
because "a multi-queue driver for Ceph" is really vague.
Thanks,
Ilya
17 Jul
17 Jul
2:19 a.m.
Hi,
I completely agree to what you said regarding Ceph client. This is exactly
my understanding of a Ceph client.
And regarding blk-mq, I meant for a block device. A multi-queue
implementation of a block device.
On Wednesday, July 15, 2020, Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Wed, Jul 15, 2020 at 12:47 AM Bobby
<italienisch1987(a)gmail.com> wrote:
>
>
>
> Hi Ilya,
>
> Thanks for the reply. It's basically both i.e. I have a specific project
currently and also I am looking to make ceph-fuse faster.
>
> But for now, let me ask specifically the project based question. In the
project
I have to write a blk-mq kernel driver for the Ceph client machine.
The Ceph client machine will transfer the data to HBA or lets say any
embedded device.
What is a "Ceph client machine"?
A Ceph client (or more specifically a RADOS client) speaks RADOS
protocol and transfers data to OSD daemons. It can't transfer data
directly to a physical device because something has to take care of
replication, ensure consistency and self healing, etc. This is the
job of the OSD.
>
> My hope is that there can be an alternative and that alternative is to
not
implement a blk-mq kernel driver and instead do the stuff in userspace.
I am trying to avoid writing a blk-mq kernel driver and yet achieve the
multi-queue implementation through userspace. Is it possible?
>
> Also AFAIK, the Ceph’s block storage implementation uses a client module
and
this client module has two implementations librbd (user-space) and krbd
(kernel module). I have not gone deep into these client modules. but can
librbd help me with this?
I guess I don't understand the goal of your project. A multi-queue
implementation of what exactly? A Ceph block device, a Ceph filesystem
or something else entirely? It would help if you were more specific
because "a multi-queue driver for Ceph" is really vague.
Thanks,
Ilya
3:11 a.m.
On Thu, Jul 16, 2020 at 3:19 PM Bobby <italienisch1987(a)gmail.com> wrote:
Hi,
I completely agree to what you said regarding Ceph client. This is exactly my
understanding of a Ceph client.
And regarding blk-mq, I meant for a block device. A multi-queue implementation of a block
device.
krbd is a blk-mq implementation of a block device. So is the nbd block
device driver -- which can be combined w/ rbd-nbd (or any other NBD
server) to utilize librbd in user-space.
On Wednesday, July 15, 2020, Ilya Dryomov
<idryomov(a)gmail.com> wrote:
--
Jason
On Wed, Jul 15, 2020 at 12:47 AM Bobby
<italienisch1987(a)gmail.com> wrote:
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
Hi Ilya,
Thanks for the reply. It's basically both i.e. I have a specific project currently
and also I am looking to make ceph-fuse faster.
But for now, let me ask specifically the project based question. In the project I have to
write a blk-mq kernel driver for the Ceph client machine. The Ceph client machine will
transfer the data to HBA or lets say any embedded device.
What is a "Ceph client machine"?
A Ceph client (or more specifically a RADOS client) speaks RADOS
protocol and transfers data to OSD daemons. It can't transfer data
directly to a physical device because something has to take care of
replication, ensure consistency and self healing, etc. This is the
job of the OSD.
My hope is that there can be an alternative and that alternative is to not implement a
blk-mq kernel driver and instead do the stuff in userspace. I am trying to avoid writing a
blk-mq kernel driver and yet achieve the multi-queue implementation through userspace. Is
it possible?
Also AFAIK, the Ceph’s block storage implementation uses a client module and this client
module has two implementations librbd (user-space) and krbd (kernel module). I have not
gone deep into these client modules. but can librbd help me with this?
I guess I don't understand the goal of your project. A multi-queue
implementation of what exactly? A Ceph block device, a Ceph filesystem
or something else entirely? It would help if you were more specific
because "a multi-queue driver for Ceph" is really vague.
Thanks,
Ilya
_______________________________________________
1387
days inactive
1390
days old
5 comments
3 participants
participants (3)
-
Bobby -
Ilya Dryomov -
Jason Dillaman