Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754534AbdLNX7q (ORCPT ); Thu, 14 Dec 2017 18:59:46 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53840 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754458AbdLNX7k (ORCPT ); Thu, 14 Dec 2017 18:59:40 -0500 Subject: Re: Setting large MTU size on slave interfaces may stall the whole system To: Or Gerlitz Cc: Linux Netdev List , Linux Kernel , Jay Vosburgh , Veaceslav Falico , Andy Gospodarek , Aviv Heller , Moni Shoua , Mahesh Bandewar , "David S. Miller" References: <9f95c2a0-e4fe-270d-790a-beeb6b3e7690@oracle.com> From: Qing Huang Message-ID: <91b6e7f4-e1f5-fe9c-f694-91b9f62e1608@oracle.com> Date: Thu, 14 Dec 2017 16:00:05 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8745 signatures=668648 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=883 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1712140326 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1590 Lines: 43 Hi Or, On 12/13/2017 06:28 AM, Or Gerlitz wrote: > On Tue, Dec 12, 2017 at 5:21 AM, Qing Huang wrote: >> Hi, >> >> We found an issue with the bonding driver when testing Mellanox devices. >> The following test commands will stall the whole system sometimes, with >> serial console >> flooded with log messages from the bond_miimon_inspect() function. Setting >> mtu size >> to be 1500 seems okay but very rarely it may hit the same problem too. >> >> ip address flush dev ens3f0 >> ip link set dev ens3f0 down >> ip address flush dev ens3f1 >> ip link set dev ens3f1 down >> [root@ca-hcl629 etc]# modprobe bonding mode=0 miimon=250 use_carrier=1 >> updelay=500 downdelay=500 >> [root@ca-hcl629 etc]# ifconfig bond0 up >> [root@ca-hcl629 etc]# ifenslave bond0 ens3f0 ens3f1 >> [root@ca-hcl629 etc]# ip link set bond0 mtu 4500 up >> Seiral console output: >> >> ** 4 printk messages dropped ** [ 3717.743761] bond0: link status down for >> interface ens3f0, disabling it in 500 ms > [..] > > >> It seems that when setting a large mtu size on an RoCE interface, the RTNL >> mutex may be held too long by the slave >> interface, causing bond_mii_monitor() to be called repeatedly at an interval >> of 1 tick (1K HZ kernel configuration) and kernel to become unresponsive. > Did you try/managed to reproduce that also with other NIC drivers? This seems to be an issue with the bonding driver. Also running older kernels on the same hardware without commit (de77ecd4ef02) seems to work fine. We can try to reproduce the issue with other type of NIC hardware. > > Or.