Hi Ilya, Thanks for the reply. It's basically both i.e. I have a specific project currently and also I am looking to make ceph-fuse faster. But for now, let me ask specifically the project based question. In the project I have to write a blk-mq kernel driver for the Ceph client machine. The Ceph client machine will transfer the data to HBA or lets say any embedded device. My hope is that there can be an alternative and that alternative is to not implement a blk-mq kernel driver and instead do the stuff in userspace. I am trying to avoid writing a blk-mq kernel driver and yet achieve the multi-queue implementation through userspace. Is it possible? Also AFAIK, the Ceph’s block storage implementation uses a client module and this client module has two implementations librbd (user-space) and krbd (kernel module). I have not gone deep into these client modules. but can librbd help me with this? Thanks. Looking forward to your reply :-) BR On Tue, Jul 14, 2020 at 11:16 PM Ilya Dryomov <idryomov@gmail.com> wrote:

On Mon, Jul 13, 2020 at 8:40 PM Bobby <italienisch1987@gmail.com> wrote:

...

Hi,

I have a question regarding support for multiple BLK-MQ queues for

Ceph's RADOS Block Device (RBD). The below given link says that the driver has been using the BLK-MQ interface for a while but not actually multiple queues until now with having a queue per-CPU. A change to not hold onto caps that aren't actually needed. These improvements and more can be found as part of the Ceph changes for Linux 5.7, which should be released as stable in early June.

...

https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Performance

...

My question is: Is it possible that through Ceph FS (Filesystem in User

Space) I can develop a multi-queue driver for Ceph? Asking because this way I can avoid kernel space. ( https://docs.ceph.com/docs/nautilus/start/quick-cephfs/)

[ trimming CCs to dev and ceph-devel ]

Hi Bobby,

I'm not sure what you mean by a "multi-queue driver for Ceph". blk-mq is the block layer framework, it has nothing to do with filesystems, whether local sitting on top of a block device (such as ext4 or XFS) or distributed sitting on top of a network stack (such as CephFS).

Do you have a specific project in mind or are you just looking to make ceph-fuse faster?

Thanks,

Ilya

On Wed, Jul 15, 2020 at 12:47 AM Bobby <italienisch1987@gmail.com> wrote:

...

Hi Ilya,

Thanks for the reply. It's basically both i.e. I have a specific project

currently and also I am looking to make ceph-fuse faster.

...

But for now, let me ask specifically the project based question. In the

Hi, I completely agree to what you said regarding Ceph client. This is exactly my understanding of a Ceph client. And regarding blk-mq, I meant for a block device. A multi-queue implementation of a block device. On Wednesday, July 15, 2020, Ilya Dryomov <idryomov@gmail.com> wrote: project I have to write a blk-mq kernel driver for the Ceph client machine. The Ceph client machine will transfer the data to HBA or lets say any embedded device.

What is a "Ceph client machine"?

A Ceph client (or more specifically a RADOS client) speaks RADOS protocol and transfers data to OSD daemons. It can't transfer data directly to a physical device because something has to take care of replication, ensure consistency and self healing, etc. This is the job of the OSD.

...

My hope is that there can be an alternative and that alternative is to

...

Also AFAIK, the Ceph’s block storage implementation uses a client module

and this client module has two implementations librbd (user-space) and krbd (kernel module). I have not gone deep into these client modules. but can

not implement a blk-mq kernel driver and instead do the stuff in userspace. I am trying to avoid writing a blk-mq kernel driver and yet achieve the multi-queue implementation through userspace. Is it possible? librbd help me with this?

I guess I don't understand the goal of your project. A multi-queue implementation of what exactly? A Ceph block device, a Ceph filesystem or something else entirely? It would help if you were more specific because "a multi-queue driver for Ceph" is really vague.

Thanks,

Ilya