Questions on Ceph on ARM
Hi all, I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster?
It should be fine. I use arm64 systems as clients, and would expect them to be fine for servers. The biggest problem would be performance. ~ Sean On Jul 6, 2020, 5:04 AM -0500, norman <norman.kern@gmx.com>, wrote:
Hi all,
I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sean, Thanks for your reply. I will test the perf on ARM. On 7/7/2020 上午7:41, Sean Johnson wrote:
It should be fine. I use arm64 systems as clients, and would expect them to be fine for servers. The biggest problem would be performance.
~ Sean On Jul 6, 2020, 5:04 AM -0500, norman <norman.kern@gmx.com>, wrote:
Hi all,
I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just for Info : Can you give us some info, which Hardware and Operating System are you using to run arm64 ? Br, Am 07.07.2020 um 01:41 schrieb Sean Johnson:
It should be fine. I use arm64 systems as clients, and would expect them to be fine for servers. The biggest problem would be performance.
~ Sean On Jul 6, 2020, 5:04 AM -0500, norman <norman.kern@gmx.com>, wrote:
Hi all,
I'm using Ceph on X86 and ARM, is it safe make x86 and arm64 in the same cluster? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Siegfried Höllrigl Technik ________________________________________________________________________________ Xidras GmbH Stockern 47 3744 Stockern Austria Tel: +43 (0)2983 201 30505 Fax: +43 (0)2983 201 30505 9 Email: siegfried.hoellrigl@xidras.com Web: http://www.xidras.com FN 317036 f | Landesgericht Krems | ATU64485024 ________________________________________________________________________________ VERTRAULICHE INFORMATIONEN! Diese eMail enthält vertrauliche Informationen und ist nur für den berechtigten Empfänger bestimmt. Wenn diese eMail nicht für Sie bestimmt ist, bitten wir Sie, diese eMail an uns zurückzusenden und anschließend auf Ihrem Computer und Mail-Server zu löschen. Solche eMails und Anlagen dürfen Sie weder nutzen, noch verarbeiten oder Dritten zugänglich machen, gleich in welcher Form. Wir danken für Ihre Kooperation! CONFIDENTIAL! This email contains confidential information and is intended for the authorised recipient only. If you are not an authorised recipient, please return the email to us and then delete it from your computer and mail-server. You may neither use nor edit any such emails including attachments, nor make them accessible to third parties in any manner whatsoever. Thank you for your cooperation
Hi Norman, It works well. We mix Arm and x86. For example, OSD and MON are on Arm, RGWs are on x86. We can also put x86 OSD in the same cluster. All of the Ceph daemons talk to each other with the same protocol. Just separate the OSD to different CRUSH root If the OSD on Arm and x86 has a significant performance difference. Best regards, Aaron
Aaron, Significant performance diff for OSD? Can you tell me percentage of perf descent? Best regards, Norman On 8/7/2020 上午9:56, Aaron Joue wrote:
Hi Norman,
It works well. We mix Arm and x86. For example, OSD and MON are on Arm, RGWs are on x86. We can also put x86 OSD in the same cluster. All of the Ceph daemons talk to each other with the same protocol. Just separate the OSD to different CRUSH root If the OSD on Arm and x86 has a significant performance difference.
Best regards, Aaron _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Bear in mind that ARM and x86 are architectures, not CPU models. Both are available in a vast variety of core counts, clocks, and implementations. Eg., an 80 core Ampere Altra likely will smoke a Intel Atom D410 in every way. That said, what does “performance” mean? For object storage, it might focus on throughput; for block attached to VM instances, chances are that latency (iops) is the dominant concern. And if you’re running LFF HDDs, it’s less likely to matter than if your nodes are graced with an abundance of NVMe devices split into multiple OSDs and using dmcrypt. You might like to visit softiron.com if they aren’t already on your radar. If they fit your use-case, their ARM servers boast really low power usage compared to eg. a dual Xeon chassis: less power, less cooling, RUs become the limiting factor in your racks instead of ahmps, etc. — me
Aaron,
Significant performance diff for OSD? Can you tell me percentage of perf descent?
Best regards,
Norman
On 8/7/2020 上午9:56, Aaron Joue wrote:
Hi Norman,
It works well. We mix Arm and x86. For example, OSD and MON are on Arm, RGWs are on x86. We can also put x86 OSD in the same cluster. All of the Ceph daemons talk to each other with the same protocol. Just separate the OSD to different CRUSH root If the OSD on Arm and x86 has a significant performance difference.
Best regards, Aaron _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Anthony, I just used normal HDD, I pretend to test the same HDDs on two X86 and ARM clusters to test the cephfs perf diff. Best regards, Norman On 8/7/2020 上午11:51, Anthony D'Atri wrote:
Bear in mind that ARM and x86 are architectures, not CPU models. Both are available in a vast variety of core counts, clocks, and implementations. Eg., an 80 core Ampere Altra likely will smoke a Intel Atom D410 in every way.
That said, what does “performance” mean? For object storage, it might focus on throughput; for block attached to VM instances, chances are that latency (iops) is the dominant concern.
And if you’re running LFF HDDs, it’s less likely to matter than if your nodes are graced with an abundance of NVMe devices split into multiple OSDs and using dmcrypt.
You might like to visit softiron.com if they aren’t already on your radar. If they fit your use-case, their ARM servers boast really low power usage compared to eg. a dual Xeon chassis: less power, less cooling, RUs become the limiting factor in your racks instead of ahmps, etc.
— me
Aaron,
Significant performance diff for OSD? Can you tell me percentage of perf descent?
Best regards,
Norman
On 8/7/2020 上午9:56, Aaron Joue wrote:
Hi Norman,
It works well. We mix Arm and x86. For example, OSD and MON are on Arm, RGWs are on x86. We can also put x86 OSD in the same cluster. All of the Ceph daemons talk to each other with the same protocol. Just separate the OSD to different CRUSH root If the OSD on Arm and x86 has a significant performance difference.
Best regards, Aaron _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Norman There is no fixed percentage for that. If you mix slow and fast OSD in a PG, the overall performance of the pool will be affected by the slower devices. This is the same reason as you will not mix SSD and HDD OSD in a pool. If the performance difference between OSD is small, it will be OK to mix. It is also the same recommendation if you mix different performances of any x86 servers. Best regards, Aaron
Aaron, It's the same consideration I will take, If mix them, I will worry about the jitter of perf. Best regards, Norman On 8/7/2020 下午1:33, Aaron Joue wrote:
Hi Norman There is no fixed percentage for that. If you mix slow and fast OSD in a PG, the overall performance of the pool will be affected by the slower devices. This is the same reason as you will not mix SSD and HDD OSD in a pool. If the performance difference between OSD is small, it will be OK to mix. It is also the same recommendation if you mix different performances of any x86 servers.
Best regards, Aaron _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Aaron Joue
-
Anthony D'Atri
-
norman
-
Sean Johnson
-
Siegfried Höllrigl