From:   Joakim Zhang <qiangqing.zhang@nxp.com>
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
CC:     "trond.myklebust@hammerspace.com" <trond.myklebust@hammerspace.com>,
        "anna.schumaker@netapp.com" <anna.schumaker@netapp.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "yoshfuji@linux-ipv6.org" <yoshfuji@linux-ipv6.org>,
        "dsahern@kernel.org" <dsahern@kernel.org>,
        "kuba@kernel.org" <kuba@kernel.org>, Po Liu <po.liu@nxp.com>,
        Vladimir Oltean <vladimir.oltean@nxp.com>,
        Jun Li <jun.li@nxp.com>
Subject: NFS break if unplug one when connect two cables 
Thread-Topic: NFS break if unplug one when connect two cables 
Thread-Index: AdfboN5KL9qsyUTWSD+v/jdc9YvrAA==
Date:   Wed, 17 Nov 2021 11:13:27 +0000
Message-ID: <DB8PR04MB679527531E203AE5B3A8FEFAE69A9@DB8PR04MB6795.eurprd04.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?JhA9bUEOkRISEWS1XEi7fi2MLSjD3FSFKuUHzg+wdS/Lg2RWzDEhQW0SinFv?=
 =?us-ascii?Q?ueWuqMTe+jb3GI8kp/vHF8gjbqFm2j8qIb3dhLeakpG4Db12el4JJjR6aNXG?=
 =?us-ascii?Q?7s6LHU6wj6T2ZNjm1VZZnjyIVIRylDzYk1ISxVAp9yIMjJx6/V56S5pOrJvu?=
 =?us-ascii?Q?s+B4hMBDcRtMM5x2Uc1z+YPrM6uRqezul/D0FOMFnG29fcKeCaSJoBkt3a3N?=
 =?us-ascii?Q?s4q2mRHynfHcHtnPPC/gWcpvl7ZZnwqbfgbGpIzDc2Hf81QILtIWq5wDJ3Ew?=
 =?us-ascii?Q?2Bm1qElR26Rx4IOKJuVnE+aegtsMc9bGzFUB0QAVHOWPT4qfh9j1d5zssaRL?=
 =?us-ascii?Q?xDJDkPWjllAWnzcM4acvVujPi8803qgi6zBm5pUl4pt4aA0u/9aI1ggeCKrE?=
 =?us-ascii?Q?kQj1MuYOeyf5AQEskRyNT+Ob3mNk62LFAvO50FKDQ4Fac2Ch2dSa97IML0H+?=
 =?us-ascii?Q?9InvsiYwQjPg3aoSpON3WK9Sqkz6hyMV2iye7PPYj4NzvLYLkTWm0YPWZhSm?=
 =?us-ascii?Q?0u2f8Iqq+RECmunEirw+X8qT/azL2Pm2U4ZexAOndxmh38cxpHOICENv+0h5?=
 =?us-ascii?Q?MsGbMGhUUfNbUVUIfAnYSrmvyf70E7ggkeMIxrPw0WYogWuW+O2BJN3X/xdc?=
 =?us-ascii?Q?nwK5y9IeDpLQd3TcNtDHuhfdWwx8bMV+UvRQun2bzY/eym6ffMTNsz0ig2eB?=
 =?us-ascii?Q?QduscrUyFFJMzwTOZ2NK1gYR7YJU1b4TKsoKbW6tIMO1JYy3LUWGPoGqJe2n?=
 =?us-ascii?Q?bJyn4PsT3pFqHLbxEhZoOsbCGUo+ZMAyeXUGrjWbLNVNJiulNPECtKsGvNAN?=
 =?us-ascii?Q?hp3iUhVP6y+IdQokt4bGjT5Al8VukwciSCQ1u+FuO7vG35exdNTTTF4I6UUe?=
 =?us-ascii?Q?LURRqfR7fdTyVqlsBeKJmInevauZcjXhi+GzRWZdaapVKjgN3638bBrnUBQh?=
 =?us-ascii?Q?c08cnDiSvqsNuZA3NTeT1pi36ynzk9ZUx9S42Nt3GBQNNWsH7NHIkLXtol54?=
 =?us-ascii?Q?gbW7QfS7Ey7eIYtm3rAt+R0/TQHNVzU9DF2rVHwHhNrWpZjRiSJuZKQ+PZFV?=
 =?us-ascii?Q?U3DF7aX0pKEJbFMLbWWznbFFP0JguXWC2blEDoH//SM/7k8pHra06wafhVkZ?=
 =?us-ascii?Q?OLhXQPLlZLGYQRvWV0ejp8TkCEmo4SjeW1m1aVkfMI0ZoQ1/fTBYJycoUKL/?=
 =?us-ascii?Q?XxLfoOHYE0Exmu9UeAFUFujvFXKSQF68sCs6Zf2vo/rMsnhnMWwbcDH4XNcO?=
 =?us-ascii?Q?w1DYDA5XSc3i0lx2NpdbLVpPX9bPGoKTtVLMdQTyEdAdT9KzvdNGoUFeOmWe?=
 =?us-ascii?Q?W8m3RW59B1awlj1DHl2479ZEZqwSzW4hdUuXgZtrZ4SUFBZyvKU7hhTyiicM?=
 =?us-ascii?Q?NfKZONu7PSSs4JJRKgAukKFQxJW8/Iuff9Eqd8S+uKnnINdvx8+WUmZ5RgYH?=
 =?us-ascii?Q?C2PmXigywmzkbRHCpRIKknTK+iCAcW3jDHLrXzpCdKKNUIjQ0piIbjWu/2ow?=
 =?us-ascii?Q?BsKb9/MPu7JPjn7x4TobZoPB+nMk72QHEbtVq6O3SKkQURD5JTYknl93Atk4?=
 =?us-ascii?Q?LLOLkOMKc6PssDFepDMcgopSmRMJnuaD+TUfxa7/cqK2LBvPlHS7XoeDJ7Q/?=
 =?us-ascii?Q?Zg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DB8PR04MB6795.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 210be4f2-43a5-4b59-d56c-08d9a9bb47b1
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2021 11:13:27.5266
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: OiowcOCuoYEBP7m60IkZ4WD6plZKMbSNTbY2NNbqkOPibdy/o1G2ysv5ZOc1DgOCczO/KrKPsTiuN8OT1X47Rg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR04MB3207
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


Hi community guys,

I encountered a NFS issue at my side, which bothering me much, could you he=
lp me if you have some ideas? That would be appreciated if you have some in=
sights. :-)

---------------------------------------------------------------------------=
---------------------------------------------------------------------------=
---------------------------------------
Environment:
	1) A board with two ethernet controllers, eth0 is FEC, eth1 is STMMAC.
	2) plugged two cables in the same router.
	3) NFS is Yocto.

The kernel boot log, we can see that NFS mount via eth0.
[    8.860717] Sending DHCP requests ., OK
[    9.009096] IP-Config: Got DHCP answer from 10.193.102.254, my address i=
s 10.193.102.150
[    9.017377] IP-Config: Complete:
[    9.020737]      device=3Deth0, hwaddr=3D00:04:9f:06:e2:97, ipaddr=3D10.=
193.102.150, mask=3D255.255.255.0, gw=3D10.193.102.254
[    9.032123]      host=3D10.193.102.150, domain=3Dap.freescale.net, nis-d=
omain=3D(none)
[    9.039614]      bootserver=3D0.0.0.0, rootserver=3D10.193.108.176, root=
path=3D
[    9.039625]      nameserver0=3D165.114.89.4, nameserver1=3D134.27.184.42

After NFS mounted, I dump the below info:
~# cat /proc/cmdline
console=3Dttymxc1,115200 root=3D/dev/nfs ip=3Ddhcp nfsroot=3D10.193.108.176=
:/home/nfsroot/imx8mpevk,v3,tcp

~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group d=
efault qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group=
 default qlen 1000
    link/ether 00:04:9f:06:e2:97 brd ff:ff:ff:ff:ff:ff
    inet 10.193.102.150/24 brd 10.193.102.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::204:9fff:fe06:e297/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc mq state =
UP group default qlen 1000
    link/ether 00:04:9f:06:e2:98 brd ff:ff:ff:ff:ff:ff
    inet 10.193.102.85/24 brd 10.193.102.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:9fff:fe06:e298/64 scope link
       valid_lft forever preferred_lft forever
4: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can

~# ip route
default via 10.193.102.254 dev eth0
10.193.102.0/24 dev eth0 proto kernel scope link src 10.193.102.150
10.193.102.0/24 dev eth1 proto kernel scope link src 10.193.102.85
10.193.102.254 dev eth1 scope link
134.27.184.42 via 10.193.102.254 dev eth1
165.114.89.4 via 10.193.102.254 dev eth1

The issue is:
	1) unplug the eth1 then re-plug, NFS can't turn back.
	2) unplug the eth0 then re-plug, NFS can turn back.
My queston is that why unplug the eth1 would break the NFS? Is it a NFS lim=
itation?

---------------------------------------------------------------------------=
---------------------------------------------------------------------------=
----------------------------------------------
To figure out it, I tried below two approaches.

1) As you can see, eth0 and eth1 are in the same subnet, may this lead to t=
his issue? Firstly I connect eth1 to another board. After NFS mounted,
all is well, but minitues later, the NFS break, I supposed the reason is et=
h1 get a local ip which belong to another subnet. So what I can conclude
here is that, NFS still can work if eth0 and eth1 are in the same subnet, m=
eanwhile I didn't unplug eth1, but for different subnets, NFS can't work di=
rectly.
So this issue is not related to the same subnet.

2) I thought it should not be a kernel issue, may caused by userspace, so I=
 disable the network service.
	~# systemctl disable systemd-networkd.service
	~# systemctl disable connman.service
  I repeated the operation, after NFS mounted, only eth0 is up since I disa=
ble the network service, so I manually up the eth1 and use udhcpc to get th=
e ip.

~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group d=
efault qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group=
 default qlen 1000
    link/ether 00:04:9f:06:e2:97 brd ff:ff:ff:ff:ff:ff
    inet 10.193.102.53/24 brd 10.193.102.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::204:9fff:fe06:e297/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group=
 default qlen 1000
    link/ether 00:04:9f:06:e2:98 brd ff:ff:ff:ff:ff:ff
    inet 10.193.102.85/24 brd 10.193.102.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:9fff:fe06:e298/64 scope link
       valid_lft forever preferred_lft forever

~# ip route
default via 10.193.102.254 dev eth0
default via 10.193.102.254 dev eth1 metric 10
10.193.102.0/24 dev eth0 proto kernel scope link src 10.193.102.53
10.193.102.0/24 dev eth1 proto kernel scope link src 10.193.102.85

Now I unplug the eth1, the NFS still can work. For this experiment, this is=
sue seems related to network service.
What I suspected is that route changed by network service when unplug the e=
th1 or the original route has some limitation?
I compare the route with and w/o network service enabled.

Network service enabled:
~# ip route
default via 10.193.102.254 dev eth0
10.193.102.0/24 dev eth0 proto kernel scope link src 10.193.102.150
10.193.102.0/24 dev eth1 proto kernel scope link src 10.193.102.85
10.193.102.254 dev eth1 scope link
134.27.184.42 via 10.193.102.254 dev eth1
165.114.89.4 via 10.193.102.254 dev eth1

Network service disabled:
~# ip route
default via 10.193.102.254 dev eth0
default via 10.193.102.254 dev eth1 metric 10
10.193.102.0/24 dev eth0 proto kernel scope link src 10.193.102.53
10.193.102.0/24 dev eth1 proto kernel scope link src 10.193.102.85

Why only one route item towards "default" when network service enabled, but=
 there are two items when network service disabled?

I have little knowledge about this scope, do any one know more?

Best Regards,
Joakim Zhang