All; I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible. We're talking 1/4 to 1/3 of our current solution. I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client). What can I look at to tune this, preferably on the iSCSI gateways? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc: https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv... Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained. Mark On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Mark. I’m interested as well, wanting to provide block service to baremetal hosts; iSCSI seems to be the classic way to do that. I know there’s some work on MS Windows RBD code, but I’m uncertain if it’s production-worthy, and if RBD namespaces suffice for tenant isolation — and are themselves mature. Thoughts anyone?
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Not my area of expertise I'm afraid. I did most of this testing when I was adding the "client endpoints" support to CBT so we could use the same fio benchmark code across the whole range of ceph block/fs clients. One of the RBD guys might be able to answer your questions though! If it were me, I think I would stick with kernel rbd if possible. It generally performs well with lower overhead (though in our lab we see a ~3GB/s limit per kernel client that we haven't yet been able to explain. librbd was able to operate at 6-8GB/s+). Mark On 10/5/20 5:55 PM, Anthony D'Atri wrote:
Thanks, Mark.
I’m interested as well, wanting to provide block service to baremetal hosts; iSCSI seems to be the classic way to do that.
I know there’s some work on MS Windows RBD code, but I’m uncertain if it’s production-worthy, and if RBD namespaces suffice for tenant isolation — and are themselves mature.
Thoughts anyone?
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Mark, Why do you use io_depth=32 in fio parameters? Is there any reason for not choose 16 or 64? Thanks in advance!
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To be honest I don't really remember, those tests were from a while ago. :) I'm guessing I probably was getting higher throughput with 32 vs 16 in some of the test cases but didn't need to go up to 64 at that time. This was all before various work we've done in bluestore over the past year that's improved performance quite a bit in our rbd tests. Mark On 10/5/20 6:08 PM, Tecnología CHARNE.NET wrote:
Mark, Why do you use io_depth=32 in fio parameters?
Is there any reason for not choose 16 or 64?
Thanks in advance!
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Mark; Are you suggesting some other means to configure iSCSI targets with Ceph? If so, how do configure for non-tcmu? The iSCSI clients are not RBD aware, and I can't really make them RBD aware. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Mark Nelson [mailto:mnelson@redhat.com] Sent: Monday, October 5, 2020 3:40 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph iSCSI Performance I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc: https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv... Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained. Mark On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dominic, If you can't use kernel rbd I think you'll probably have to deal with the higher overhead and lower performance with the tcmu solution. It's possible there might be some things you can tweak at the tcmu layer that will improve things, but when I looked at it there simply seemed to be a lot of extra work being done to do the translation. YMMV. Mark On 10/6/20 12:49 PM, DHilsbos@performair.com wrote:
Mark;
Are you suggesting some other means to configure iSCSI targets with Ceph?
If so, how do configure for non-tcmu?
The iSCSI clients are not RBD aware, and I can't really make them RBD aware.
Thank you,
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Mark Nelson [mailto:mnelson@redhat.com] Sent: Monday, October 5, 2020 3:40 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph iSCSI Performance
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can try PetaSAN www.petasan.org we use rbd backend by SUSE. It works out of the box. /Maged On 06/10/2020 19:49, DHilsbos@performair.com wrote:
Mark;
Are you suggesting some other means to configure iSCSI targets with Ceph?
If so, how do configure for non-tcmu?
The iSCSI clients are not RBD aware, and I can't really make them RBD aware.
Thank you,
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Mark Nelson [mailto:mnelson@redhat.com] Sent: Monday, October 5, 2020 3:40 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph iSCSI Performance
I don't have super recent results, but we do have some test data from last year looking at kernel rbd, rbd-nbd, rbd+tcmu, fuse, etc:
https://docs.google.com/spreadsheets/d/1oJZ036QDbJQgv2gXts1oKKhMOKXrOI2XLTkv...
Generally speaking going through the tcmu layer was slower than kernel rbd or librbd directly (sometimes by quite a bit!). There was also more client side CPU usage per unit performance as well (which makes sense since there's additional work being done). You may be able to get some of that performance back with more clients as I do remember there being some issues with iodepth and tcmu. The only setup that I remember being slower at the time though was rbd-fuse which I don't think is even really maintained.
Mark
On 10/5/20 4:43 PM, DHilsbos@performair.com wrote:
All;
I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible.
We're talking 1/4 to 1/3 of our current solution.
I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client).
What can I look at to tune this, preferably on the iSCSI gateways?
Thank you,
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
DHilsbos＠performair.com
-
Maged Mokhtar
-
Mark Nelson
-
Tecnología CHARNE.NET