Ceph Nautius not working after setting MTU 9000
Hi, I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out. regards Amudhan P
On 5/23/20 12:02 PM, Amudhan P wrote:
Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this. Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets. This is definitely not a Ceph issue. Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I have set Network switch with MTU size 9000 and also in my netplan configuration. What else needs to be checked? On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote:
Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In OSD logs "heartbeat_check: no reply from OSD" On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote:
Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can the servers/nodes ping eachother using large packet sizes? I guess not. Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote: Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices. If you have just one node which it not correctly configured for MTU 9000 it wouldn't work. On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote: Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Please check you MTU limit at the switch level, cand check other ressources with icmp ping. Try to add 14Byte for ethernet header at your switch level mean an MTU of 9014 ? are you using juniper ??? Exemple : ping -D -s 90000 other_ip Le sam. 23 mai 2020 à 15:18, Khodayar Doustar <doustar@rayanexon.ir> a écrit :
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote: Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, It should be ping -M do -s 8972 IP ADDRESS. You can't ping with 9000 size. If you can't ping with 8972 size, then somewhere in the path MTU config is wrong. Regards, Suresh On Sat, May 23, 2020, 1:35 PM apely agamakou <moodymob@gmail.com> wrote:
Hi,
Please check you MTU limit at the switch level, cand check other ressources with icmp ping. Try to add 14Byte for ethernet header at your switch level mean an MTU of 9014 ? are you using juniper ???
Exemple : ping -D -s 90000 other_ip
Le sam. 23 mai 2020 à 15:18, Khodayar Doustar <doustar@rayanexon.ir> a écrit :
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
> On 5/23/20 12:02 PM, Amudhan P wrote: > Hi, > > I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 > (default) recently i tried to update MTU size to 9000. > After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
> > regards > Amudhan P > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
it's a Dell S4048T-ON Switch using 10G Ethernet. On Sat, May 23, 2020 at 11:05 PM apely agamakou <moodymob@gmail.com> wrote:
Hi,
Please check you MTU limit at the switch level, cand check other ressources with icmp ping. Try to add 14Byte for ethernet header at your switch level mean an MTU of 9014 ? are you using juniper ???
Exemple : ping -D -s 90000 other_ip
Le sam. 23 mai 2020 à 15:18, Khodayar Doustar <doustar@rayanexon.ir> a écrit :
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
> On 5/23/20 12:02 PM, Amudhan P wrote: > Hi, > > I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 > (default) recently i tried to update MTU size to 9000. > After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
> > regards > Amudhan P > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, I have set setting on the switch side also. On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
On 5/23/20 12:02 PM, Amudhan P wrote: Hi,
I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
regards Amudhan P _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Does your ping work or not? On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote:
> On 5/23/20 12:02 PM, Amudhan P wrote: > Hi, > > I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 > (default) recently i tried to update MTU size to 9000. > After setting Jumbo frame running ceph -s is timing out.
Ceph can run just fine with an MTU of 9000. But there is probably something else wrong on the network which is causing this.
Check the Jumbo Frames settings on all the switches as well to make sure they forward all the packets.
This is definitely not a Ceph issue.
Wido
> > regards > Amudhan P > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No, ping with MTU size 9000 didn't work. On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote:
Hi,
I have set Network switch with MTU size 9000 and also in my netplan configuration.
What else needs to be checked?
> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com> wrote: > > > >> On 5/23/20 12:02 PM, Amudhan P wrote: >> Hi, >> >> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >> (default) recently i tried to update MTU size to 9000. >> After setting Jumbo frame running ceph -s is timing out. > > Ceph can run just fine with an MTU of 9000. But there is probably > something else wrong on the network which is causing this. > > Check the Jumbo Frames settings on all the switches as well to make sure > they forward all the packets. > > This is definitely not a Ceph issue. > > Wido > >> >> regards >> Amudhan P >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So this is your problem, it has nothing to do with Ceph. Just fix the network or rollback all changes. On Sun, May 24, 2020 at 9:05 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote: > > Hi, > > I have set Network switch with MTU size 9000 and also in my netplan > configuration. > > What else needs to be checked? > > >> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com
wrote:
>> >> >> >>> On 5/23/20 12:02 PM, Amudhan P wrote: >>> Hi, >>> >>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >>> (default) recently i tried to update MTU size to 9000. >>> After setting Jumbo frame running ceph -s is timing out. >> >> Ceph can run just fine with an MTU of 9000. But there is probably >> something else wrong on the network which is causing this. >> >> Check the Jumbo Frames settings on all the switches as well to make sure >> they forward all the packets. >> >> This is definitely not a Ceph issue. >> >> Wido >> >>> >>> regards >>> Amudhan P >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I didn't do any changes but started working now with jumbo frames. On Sun, May 24, 2020 at 1:04 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
So this is your problem, it has nothing to do with Ceph. Just fix the network or rollback all changes.
On Sun, May 24, 2020 at 9:05 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven: > > In OSD logs "heartbeat_check: no reply from OSD" > >> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote: >> >> Hi, >> >> I have set Network switch with MTU size 9000 and also in my netplan >> configuration. >> >> What else needs to be checked? >> >> >>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < wido@42on.com> wrote: >>> >>> >>> >>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>> Hi, >>>> >>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >>>> (default) recently i tried to update MTU size to 9000. >>>> After setting Jumbo frame running ceph -s is timing out. >>> >>> Ceph can run just fine with an MTU of 9000. But there is probably >>> something else wrong on the network which is causing this. >>> >>> Check the Jumbo Frames settings on all the switches as well to make sure >>> they forward all the packets. >>> >>> This is definitely not a Ceph issue. >>> >>> Wido >>> >>>> >>>> regards >>>> Amudhan P >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ping with 9000 MTU won't get response as I said and it should be 8972. Glad it is working but you should know what happened to avoid this issue later. On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven:
In OSD logs "heartbeat_check: no reply from OSD"
> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote: > > Hi, > > I have set Network switch with MTU size 9000 and also in my netplan > configuration. > > What else needs to be checked? > > >> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com
wrote:
>> >> >> >>> On 5/23/20 12:02 PM, Amudhan P wrote: >>> Hi, >>> >>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >>> (default) recently i tried to update MTU size to 9000. >>> After setting Jumbo frame running ceph -s is timing out. >> >> Ceph can run just fine with an MTU of 9000. But there is probably >> something else wrong on the network which is causing this. >> >> Check the Jumbo Frames settings on all the switches as well to make sure >> they forward all the packets. >> >> This is definitely not a Ceph issue. >> >> Wido >> >>> >>> regards >>> Amudhan P >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just save yourself the trouble. You won't have any real benefit from MTU 9000. It has some smallish, but it is not worth the effort, problems, and loss of reliability for most environments. Try it yourself and do some benchmarks, especially with your regular workload on the cluster (not the maximum peak performance), then drop the MTU to default ;). Please if anyone has other real world benchmarks showing huge differences in regular Ceph clusters, please feel free to post it here. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama <sstkadu@gmail.com>:
Ping with 9000 MTU won't get response as I said and it should be 8972. Glad it is working but you should know what happened to avoid this issue later.
On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven: > > In OSD logs "heartbeat_check: no reply from OSD" > >> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote: >> >> Hi, >> >> I have set Network switch with MTU size 9000 and also in my netplan >> configuration. >> >> What else needs to be checked? >> >> >>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < wido@42on.com
wrote: >>> >>> >>> >>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>> Hi, >>>> >>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >>>> (default) recently i tried to update MTU size to 9000. >>>> After setting Jumbo frame running ceph -s is timing out. >>> >>> Ceph can run just fine with an MTU of 9000. But there is probably >>> something else wrong on the network which is causing this. >>> >>> Check the Jumbo Frames settings on all the switches as well to make sure >>> they forward all the packets. >>> >>> This is definitely not a Ceph issue. >>> >>> Wido >>> >>>> >>>> regards >>>> Amudhan P >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All, Regarding Martin's observations about Jumbo Frames.... I have recently been gathering some notes from various internet sources regarding Linux network performance, and Linux performance in general, to be applied to a Ceph cluster I manage but also to the rest of the Linux server farm I'm responsible for. In short, enabling Jumbo Frames without also tuning a number of other kernel and NIC attributes will not provide the performance increases we'd like to see. I have not yet had a chance to go through the rest of the testing I'd like to do, but I can confirm (via iperf3) that only enabling Jumbo Frames didn't make a significant difference. Some of the other attributes I'm referring to are incoming and outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt coalescing, NIC offload functions that should or shouldn't be turned on, packet queuing disciplines (tc), the best choice of TCP slow-start algorithms, and other TCP features and attributes. The most off-beat item I saw was something about adding IPTABLES rules to bypass CONNTRACK table lookups. In order to do anything meaningful to assess the effect of all of these settings I'd like to figure out how to set them all via Ansible - so more to learn before I can give opinions. --> If anybody has added this type of configuration to Ceph Ansible, I'd be glad for some pointers. I have started to compile a document containing my notes. It's rough, but I'd be glad to share if anybody is interested. -Dave Dave Hall Binghamton University On 5/24/2020 12:29 PM, Martin Verges wrote:
Just save yourself the trouble. You won't have any real benefit from MTU 9000. It has some smallish, but it is not worth the effort, problems, and loss of reliability for most environments. Try it yourself and do some benchmarks, especially with your regular workload on the cluster (not the maximum peak performance), then drop the MTU to default ;).
Please if anyone has other real world benchmarks showing huge differences in regular Ceph clusters, please feel free to post it here.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama <sstkadu@gmail.com>:
Ping with 9000 MTU won't get response as I said and it should be 8972. Glad it is working but you should know what happened to avoid this issue later.
On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote: > Can the servers/nodes ping eachother using large packet sizes? I guess > not. > > Sinan Polat > >> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het > volgende geschreven: >> In OSD logs "heartbeat_check: no reply from OSD" >> >>> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> > wrote: >>> Hi, >>> >>> I have set Network switch with MTU size 9000 and also in my netplan >>> configuration. >>> >>> What else needs to be checked? >>> >>> >>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < wido@42on.com > wrote: >>>> >>>> >>>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>>> Hi, >>>>> >>>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size > 1500 >>>>> (default) recently i tried to update MTU size to 9000. >>>>> After setting Jumbo frame running ceph -s is timing out. >>>> Ceph can run just fine with an MTU of 9000. But there is probably >>>> something else wrong on the network which is causing this. >>>> >>>> Check the Jumbo Frames settings on all the switches as well to make > sure >>>> they forward all the packets. >>>> >>>> This is definitely not a Ceph issue. >>>> >>>> Wido >>>> >>>>> regards >>>>> Amudhan P >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just save yourself the trouble. You won't have any real benefit from MTU 9000. It has some smallish, but it is not worth the effort, problems, and loss of reliability for most environments. Try it yourself and do some benchmarks, especially with your regular workload on the cluster (not the maximum peak performance), then drop
MTU to default ;).
Please if anyone has other real world benchmarks showing huge differences in regular Ceph clusters, please feel free to post it here.
-- Martin Verges Managing director
Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io YouTube: https://goo.gl/PGE1Bx
Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama <sstkadu@gmail.com>:
Ping with 9000 MTU won't get response as I said and it should be
I am interested. I am always setting mtu to 9000. To be honest I cannot imagine there is no optimization since you have less interrupt requests, and you are able x times as much data. Every time there something written about optimizing the first thing mention is changing to the mtu 9000. Because it is quick and easy win. -----Original Message----- From: Dave Hall [mailto:kdhall@binghamton.edu] Sent: maandag 25 mei 2020 5:11 To: Martin Verges; Suresh Rama Cc: Amudhan P; Khodayar Doustar; ceph-users Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000 All, Regarding Martin's observations about Jumbo Frames.... I have recently been gathering some notes from various internet sources regarding Linux network performance, and Linux performance in general, to be applied to a Ceph cluster I manage but also to the rest of the Linux server farm I'm responsible for. In short, enabling Jumbo Frames without also tuning a number of other kernel and NIC attributes will not provide the performance increases we'd like to see. I have not yet had a chance to go through the rest of the testing I'd like to do, but I can confirm (via iperf3) that only enabling Jumbo Frames didn't make a significant difference. Some of the other attributes I'm referring to are incoming and outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt coalescing, NIC offload functions that should or shouldn't be turned on, packet queuing disciplines (tc), the best choice of TCP slow-start algorithms, and other TCP features and attributes. The most off-beat item I saw was something about adding IPTABLES rules to bypass CONNTRACK table lookups. In order to do anything meaningful to assess the effect of all of these settings I'd like to figure out how to set them all via Ansible - so more to learn before I can give opinions. --> If anybody has added this type of configuration to Ceph Ansible, I'd be glad for some pointers. I have started to compile a document containing my notes. It's rough, but I'd be glad to share if anybody is interested. -Dave Dave Hall Binghamton University On 5/24/2020 12:29 PM, Martin Verges wrote: the 8972. Glad
it is working but you should know what happened to avoid this issue later.
On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote: > Can the servers/nodes ping eachother using large packet sizes? I guess > not. > > Sinan Polat > >> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het > volgende geschreven: >> In OSD logs "heartbeat_check: no reply from OSD" >> >>> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> > wrote: >>> Hi, >>> >>> I have set Network switch with MTU size 9000 and also in my netplan >>> configuration. >>> >>> What else needs to be checked? >>> >>> >>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < wido@42on.com > wrote: >>>> >>>> >>>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>>> Hi, >>>>> >>>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size > 1500 >>>>> (default) recently i tried to update MTU size to 9000. >>>>> After setting Jumbo frame running ceph -s is timing out. >>>> Ceph can run just fine with an MTU of 9000. But there is probably >>>> something else wrong on the network which is causing this. >>>> >>>> Check the Jumbo Frames settings on all the switches as well to make > sure >>>> they forward all the packets. >>>> >>>> This is definitely not a Ceph issue. >>>> >>>> Wido >>>> >>>>> regards >>>>> Amudhan P >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Quick and easy depends on your network infrastructure. Sometimes it is difficult or impossible to retrofit a live cluster without disruption. > On May 25, 2020, at 1:03 AM, Marc Roos <M.Roos@f1-outsourcing.eu> wrote: > > > I am interested. I am always setting mtu to 9000. To be honest I cannot > imagine there is no optimization since you have less interrupt requests, > and you are able x times as much data. Every time there something > written about optimizing the first thing mention is changing to the mtu > 9000. Because it is quick and easy win. > > > > > -----Original Message----- > From: Dave Hall [mailto:kdhall@binghamton.edu] > Sent: maandag 25 mei 2020 5:11 > To: Martin Verges; Suresh Rama > Cc: Amudhan P; Khodayar Doustar; ceph-users > Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not working > after setting MTU 9000 > > All, > > Regarding Martin's observations about Jumbo Frames.... > > I have recently been gathering some notes from various internet sources > regarding Linux network performance, and Linux performance in general, > to be applied to a Ceph cluster I manage but also to the rest of the > Linux server farm I'm responsible for. > > In short, enabling Jumbo Frames without also tuning a number of other > kernel and NIC attributes will not provide the performance increases > we'd like to see. I have not yet had a chance to go through the rest of > the testing I'd like to do, but I can confirm (via iperf3) that only > enabling Jumbo Frames didn't make a significant difference. > > Some of the other attributes I'm referring to are incoming and outgoing > buffer sizes at the NIC, IP, and TCP levels, interrupt coalescing, NIC > offload functions that should or shouldn't be turned on, packet queuing > disciplines (tc), the best choice of TCP slow-start algorithms, and > other TCP features and attributes. > > The most off-beat item I saw was something about adding IPTABLES rules > to bypass CONNTRACK table lookups. > > In order to do anything meaningful to assess the effect of all of these > settings I'd like to figure out how to set them all via Ansible - so > more to learn before I can give opinions. > > --> If anybody has added this type of configuration to Ceph Ansible, > I'd be glad for some pointers. > > I have started to compile a document containing my notes. It's rough, > but I'd be glad to share if anybody is interested. > > -Dave > > Dave Hall > Binghamton University > >> On 5/24/2020 12:29 PM, Martin Verges wrote: >> >> Just save yourself the trouble. You won't have any real benefit from > MTU >> 9000. It has some smallish, but it is not worth the effort, problems, > and >> loss of reliability for most environments. >> Try it yourself and do some benchmarks, especially with your regular >> workload on the cluster (not the maximum peak performance), then drop > the >> MTU to default ;). >> >> Please if anyone has other real world benchmarks showing huge > differences >> in regular Ceph clusters, please feel free to post it here. >> >> -- >> Martin Verges >> Managing director >> >> Mobile: +49 174 9335695 >> E-Mail: martin.verges@croit.io >> Chat: https://t.me/MartinVerges >> >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 >> Com. register: Amtsgericht Munich HRB 231263 >> >> Web: https://croit.io >> YouTube: https://goo.gl/PGE1Bx >> >> >>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama >> <sstkadu@gmail.com>: >> >>> Ping with 9000 MTU won't get response as I said and it should be > 8972. Glad >>> it is working but you should know what happened to avoid this issue > later. >>> >>>> On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote: >>> >>>> No, ping with MTU size 9000 didn't work. >>>> >>>> On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar > <doustar@rayanexon.ir> >>>> wrote: >>>> >>>>> Does your ping work or not? >>>>> >>>>> >>>>> On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> > wrote: >>>>> >>>>>> Yes, I have set setting on the switch side also. >>>>>> >>>>>> On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, > <doustar@rayanexon.ir> >>>>>> wrote: >>>>>> >>>>>>> Problem should be with network. When you change MTU it should be >>>> changed >>>>>>> all over the network, any single hup on your network should speak > and >>>>>>> accept 9000 MTU packets. you can check it on your hosts with >>> "ifconfig" >>>>>>> command and there is also equivalent commands for other >>>> network/security >>>>>>> devices. >>>>>>> >>>>>>> If you have just one node which it not correctly configured for > MTU >>>> 9000 >>>>>>> it wouldn't work. >>>>>>> >>>>>>> On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> >>> wrote: >>>>>>>> Can the servers/nodes ping eachother using large packet sizes? I >>> guess >>>>>>>> not. >>>>>>>> >>>>>>>> Sinan Polat >>>>>>>> >>>>>>>>> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> > het >>>>>>>> volgende geschreven: >>>>>>>>> In OSD logs "heartbeat_check: no reply from OSD" >>>>>>>>> >>>>>>>>>> On Sat, May 23, 2020 at 5:44 PM Amudhan P > <amudhan83@gmail.com> >>>>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have set Network switch with MTU size 9000 and also in my >>> netplan >>>>>>>>>> configuration. >>>>>>>>>> >>>>>>>>>> What else needs to be checked? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < >>> wido@42on.com >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit > MTU >>>> size >>>>>>>> 1500 >>>>>>>>>>>> (default) recently i tried to update MTU size to 9000. >>>>>>>>>>>> After setting Jumbo frame running ceph -s is timing out. >>>>>>>>>>> Ceph can run just fine with an MTU of 9000. But there is >>> probably >>>>>>>>>>> something else wrong on the network which is causing this. >>>>>>>>>>> >>>>>>>>>>> Check the Jumbo Frames settings on all the switches as well > to >>>> make >>>>>>>> sure >>>>>>>>>>> they forward all the packets. >>>>>>>>>>> >>>>>>>>>>> This is definitely not a Ceph issue. >>>>>>>>>>> >>>>>>>>>>> Wido >>>>>>>>>>> >>>>>>>>>>>> regards >>>>>>>>>>>> Amudhan P >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
Look what I have found!!! :) https://ceph.com/geen-categorie/ceph-loves-jumbo-frames/ -----Original Message----- From: Anthony D'Atri [mailto:anthony.datri@gmail.com] Sent: maandag 25 mei 2020 22:12 To: Marc Roos Cc: kdhall; martin.verges; sstkadu; amudhan83; ceph-users; doustar Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000 Quick and easy depends on your network infrastructure. Sometimes it is difficult or impossible to retrofit a live cluster without disruption. > On May 25, 2020, at 1:03 AM, Marc Roos <M.Roos@f1-outsourcing.eu> wrote: > > > I am interested. I am always setting mtu to 9000. To be honest I > cannot imagine there is no optimization since you have less interrupt > requests, and you are able x times as much data. Every time there > something written about optimizing the first thing mention is changing > to the mtu 9000. Because it is quick and easy win. > > > > > -----Original Message----- > From: Dave Hall [mailto:kdhall@binghamton.edu] > Sent: maandag 25 mei 2020 5:11 > To: Martin Verges; Suresh Rama > Cc: Amudhan P; Khodayar Doustar; ceph-users > Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not > working after setting MTU 9000 > > All, > > Regarding Martin's observations about Jumbo Frames.... > > I have recently been gathering some notes from various internet > sources regarding Linux network performance, and Linux performance in > general, to be applied to a Ceph cluster I manage but also to the rest > of the Linux server farm I'm responsible for. > > In short, enabling Jumbo Frames without also tuning a number of other > kernel and NIC attributes will not provide the performance increases > we'd like to see. I have not yet had a chance to go through the rest > of the testing I'd like to do, but I can confirm (via iperf3) that > only enabling Jumbo Frames didn't make a significant difference. > > Some of the other attributes I'm referring to are incoming and > outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt > coalescing, NIC offload functions that should or shouldn't be turned > on, packet queuing disciplines (tc), the best choice of TCP slow-start > algorithms, and other TCP features and attributes. > > The most off-beat item I saw was something about adding IPTABLES rules > to bypass CONNTRACK table lookups. > > In order to do anything meaningful to assess the effect of all of > these settings I'd like to figure out how to set them all via Ansible > - so more to learn before I can give opinions. > > --> If anybody has added this type of configuration to Ceph Ansible, > I'd be glad for some pointers. > > I have started to compile a document containing my notes. It's rough, > but I'd be glad to share if anybody is interested. > > -Dave > > Dave Hall > Binghamton University > >> On 5/24/2020 12:29 PM, Martin Verges wrote: >> >> Just save yourself the trouble. You won't have any real benefit from > MTU >> 9000. It has some smallish, but it is not worth the effort, problems, > and >> loss of reliability for most environments. >> Try it yourself and do some benchmarks, especially with your regular >> workload on the cluster (not the maximum peak performance), then drop > the >> MTU to default ;). >> >> Please if anyone has other real world benchmarks showing huge > differences >> in regular Ceph clusters, please feel free to post it here. >> >> -- >> Martin Verges >> Managing director >> >> Mobile: +49 174 9335695 >> E-Mail: martin.verges@croit.io >> Chat: https://t.me/MartinVerges >> >> croit GmbH, Freseniusstr. 31h, 81247 Munich >> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht >> Munich HRB 231263 >> >> Web: https://croit.io >> YouTube: https://goo.gl/PGE1Bx >> >> >>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama >> <sstkadu@gmail.com>: >> >>> Ping with 9000 MTU won't get response as I said and it should be > 8972. Glad >>> it is working but you should know what happened to avoid this issue > later. >>> >>>> On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote: >>> >>>> No, ping with MTU size 9000 didn't work. >>>> >>>> On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar > <doustar@rayanexon.ir> >>>> wrote: >>>> >>>>> Does your ping work or not? >>>>> >>>>> >>>>> On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> > wrote: >>>>> >>>>>> Yes, I have set setting on the switch side also. >>>>>> >>>>>> On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, > <doustar@rayanexon.ir> >>>>>> wrote: >>>>>> >>>>>>> Problem should be with network. When you change MTU it should be >>>> changed >>>>>>> all over the network, any single hup on your network should >>>>>>> speak > and >>>>>>> accept 9000 MTU packets. you can check it on your hosts with >>> "ifconfig" >>>>>>> command and there is also equivalent commands for other >>>> network/security >>>>>>> devices. >>>>>>> >>>>>>> If you have just one node which it not correctly configured for > MTU >>>> 9000 >>>>>>> it wouldn't work. >>>>>>> >>>>>>> On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> >>> wrote: >>>>>>>> Can the servers/nodes ping eachother using large packet sizes? >>>>>>>> I >>> guess >>>>>>>> not. >>>>>>>> >>>>>>>> Sinan Polat >>>>>>>> >>>>>>>>> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> > het >>>>>>>> volgende geschreven: >>>>>>>>> In OSD logs "heartbeat_check: no reply from OSD" >>>>>>>>> >>>>>>>>>> On Sat, May 23, 2020 at 5:44 PM Amudhan P > <amudhan83@gmail.com> >>>>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have set Network switch with MTU size 9000 and also in my >>> netplan >>>>>>>>>> configuration. >>>>>>>>>> >>>>>>>>>> What else needs to be checked? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < >>> wido@42on.com >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit > MTU >>>> size >>>>>>>> 1500 >>>>>>>>>>>> (default) recently i tried to update MTU size to 9000. >>>>>>>>>>>> After setting Jumbo frame running ceph -s is timing out. >>>>>>>>>>> Ceph can run just fine with an MTU of 9000. But there is >>> probably >>>>>>>>>>> something else wrong on the network which is causing this. >>>>>>>>>>> >>>>>>>>>>> Check the Jumbo Frames settings on all the switches as well > to >>>> make >>>>>>>> sure >>>>>>>>>>> they forward all the packets. >>>>>>>>>>> >>>>>>>>>>> This is definitely not a Ceph issue. >>>>>>>>>>> >>>>>>>>>>> Wido >>>>>>>>>>> >>>>>>>>>>>> regards >>>>>>>>>>>> Amudhan P >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io To >>>>>>>>>>>> unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe >>>>>>>>>>> send an email to ceph-users-leave@ceph.io >>>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe >>>>>>>>> send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe >>>>>>>> send an email to ceph-users-leave@ceph.io >>>>>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send >>>> an email to ceph-users-leave@ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an >>> email to ceph-users-leave@ceph.io >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an >> email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to ceph-users-leave@ceph.io
Amudhan, Here is a trick I've used to test and evaluate Jumbo Frames without breaking production traffic: * Open a couple root ssh sessions on each of the two systems you want to test with. o In one window start a continuous ping to the other system. * On both test systems: o Look at the output of 'ip route list'. o Use 'ip route change' to explicitly set the MTU of all of your routes to 1500. o Review the output of 'ip route list' to assure you got them all. o Use 'ip link set' to set the MTU of the interface. + Size of all packets is still 1500 due to the route MTUs + Ping should still be working like normal. * Using 'ip addr add' to create a test logical IP on each of the two systems - use something unrelated to your production IP addresses, like 2.2.2.2/24 and 2.2.2.3/24. o Go back to the procedure above and find any new routes associated with the addresses you just added. + Use 'ip addr change' to set the MTU of these new routes to 8192. o Default route not needed. * Using the test interfaces try 'ping -s 5000' or anything above 1500. o If this works, you have everything, including your network switches, set up correctly, and you're sending packets larger that 1500. o Yet your SSH sessions and all production traffic are still running at MTU 1500. o Since the route MTUs for these spare logical IPs is 8192, you should be able probe up to 'ping -s 8150' or higher, but I think you need to leave space for the IP and ICMP headers. The nice thing about this is that you haven't disrupted your production traffic at this point, and in the worst case you can undo all of these changes by rebooting the two test nodes. If you want to move your production traffic to Jumbo Frames, change the appropriate routes to MTU 8192 on all systems. Then test test test. Lastly, change your network configuration on any effected nodes so the increased MTU will be reinstated after every reboot. -Dave Dave Hall Binghamton University On 5/24/2020 9:53 AM, Suresh Rama wrote:
Ping with 9000 MTU won't get response as I said and it should be 8972. Glad it is working but you should know what happened to avoid this issue later.
On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83@gmail.com> wrote:
No, ping with MTU size 9000 didn't work.
On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar <doustar@rayanexon.ir> wrote:
Does your ping work or not?
On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83@gmail.com> wrote:
Yes, I have set setting on the switch side also.
On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, <doustar@rayanexon.ir> wrote:
Problem should be with network. When you change MTU it should be changed all over the network, any single hup on your network should speak and accept 9000 MTU packets. you can check it on your hosts with "ifconfig" command and there is also equivalent commands for other network/security devices.
If you have just one node which it not correctly configured for MTU 9000 it wouldn't work.
On Sat, May 23, 2020 at 2:30 PM sinan@turka.nl <sinan@turka.nl> wrote:
Can the servers/nodes ping eachother using large packet sizes? I guess not.
Sinan Polat
> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83@gmail.com> het volgende geschreven: > In OSD logs "heartbeat_check: no reply from OSD" > >> On Sat, May 23, 2020 at 5:44 PM Amudhan P <amudhan83@gmail.com> wrote: >> Hi, >> >> I have set Network switch with MTU size 9000 and also in my netplan >> configuration. >> >> What else needs to be checked? >> >> >>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander <wido@42on.com wrote: >>> >>> >>>> On 5/23/20 12:02 PM, Amudhan P wrote: >>>> Hi, >>>> >>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 >>>> (default) recently i tried to update MTU size to 9000. >>>> After setting Jumbo frame running ceph -s is timing out. >>> Ceph can run just fine with an MTU of 9000. But there is probably >>> something else wrong on the network which is causing this. >>> >>> Check the Jumbo Frames settings on all the switches as well to make sure >>> they forward all the packets. >>> >>> This is definitely not a Ceph issue. >>> >>> Wido >>> >>>> regards >>>> Amudhan P >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (10)
-
Amudhan P
-
Anthony D'Atri
-
apely agamakou
-
Dave Hall
-
Khodayar Doustar
-
Marc Roos
-
Martin Verges
-
sinan＠turka.nl
-
Suresh Rama
-
Wido den Hollander