Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Jankowski, Konrad0" <konrad0.jankowski@intel.com>
To:     ivecera <ivecera@redhat.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
CC:     SlawomirX Laba <slawomirx.laba@intel.com>,
        Eric Dumazet <edumazet@google.com>,
        "moderated list:INTEL ETHERNET DRIVERS" 
        <intel-wired-lan@lists.osuosl.org>,
        open list <linux-kernel@vger.kernel.org>,
        "Piotrowski, Patryk" <patryk.piotrowski@intel.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        "David S. Miller" <davem@davemloft.net>,
        "sassmann@redhat.com" <sassmann@redhat.com>
Subject: RE: [Intel-wired-lan] [PATCH net] iavf: Do not restart Tx queues
 after reset task failure
Thread-Topic: [Intel-wired-lan] [PATCH net] iavf: Do not restart Tx queues
 after reset task failure
Thread-Index: AQHY81xxx2lHPg1F40icZaVs18UPPa5EziYw
Date:   Fri, 18 Nov 2022 14:31:52 +0000
Message-ID: <DM8PR11MB5621A3E2D98C584E6C4EC1E9AB099@DM8PR11MB5621.namprd11.prod.outlook.com>
References: <20221108102502.2147389-1-ivecera@redhat.com>
In-Reply-To: <20221108102502.2147389-1-ivecera@redhat.com>
Accept-Language: pl-PL, en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?18OU0912scrQlPsi+ImLqTt+9q/unUv8t+WrSF0POjuryKpTxHzMr9CQ5uG7?=
 =?us-ascii?Q?El57dHSGeaNGxSUCut9DsXACpc8zbIspM/bpJ7LI9DqSfimXYcu/uyTEOCk4?=
 =?us-ascii?Q?69PV3i4U6hnDkuIbUZdN5H0bJOCJdmQTF4pYOe4DfmqBUYwqJOZmp6y2QFlA?=
 =?us-ascii?Q?DS6749q36euSM2eb+vALzc0xzEXpbEPusDFJezg+vBfe5JgbR2AxrYLysubj?=
 =?us-ascii?Q?FK6SAyLtJhItiYJFMwe+8v0orlvye5V6WkDycmIIE5hdcIhpPXFxpyzzIzxT?=
 =?us-ascii?Q?nmLfVH7AEVNvicFC6NWMIUaTkERStNLUpbul0dZpbL8+8wFOQRT3FwV9D5T8?=
 =?us-ascii?Q?fL9wmak0SxB/gsYByq+WYQXdFN0vkzBcxpDIf+6O2boAaknGYbXYShHxmaig?=
 =?us-ascii?Q?iS9Whla7LoH1Bk8tEerzgh0g+3FwK/RzpXBD8GD/XSTlpGGfSGc4sCjOUGLk?=
 =?us-ascii?Q?C/2rpEYS8VXHdzmCPmj7GDz82TFcC/gm5wuzteWT0/Jz90jWK584y8TsLzpD?=
 =?us-ascii?Q?krpnL0HMxU9FZlZ99W5kaZSmt+agkY91/MhBqudKuaBGNmw/UTE3h630RxMr?=
 =?us-ascii?Q?13Qe/lmhvOfbYMxPL0l3iUS5TpD0OJC5uj+qsljOwO3YGAHr32mSm9jF92B6?=
 =?us-ascii?Q?KaB3ryTtGJUk/Vulk3csI0EIqEf8FOu9yjzVIfbw+8FaTpXj5r9YaBpG2C1w?=
 =?us-ascii?Q?oM67gWzWwoDN2cwSmbq5pkGsJxd+KIykC21p/xoTN89qjjDih7gQp2MaUlRW?=
 =?us-ascii?Q?PXXQPTOzyXNPQXl/wXw1+wn9DMoM8DojaB1q7G6233FtjNTZyRs7btpsbvxd?=
 =?us-ascii?Q?5M4T2jZUsyjDOYJvnUk1iRCUkPpFZbWBtGNDBszJWVpUj0eP4I9QxXHEJD6w?=
 =?us-ascii?Q?jW8x2YllCSlIyqEbh9DAzE7dW4jXBOWOrhl/wNrJMTZadqDxAmn4Uxde94Su?=
 =?us-ascii?Q?n+28umgwsnvIbkib3q2s1QHf55cJOEOeq63n6xsPfh+Jqk9iWATzKOwErfES?=
 =?us-ascii?Q?Acs27qYpDDvLX2Iw8JKU5ngxALtWza6EH1YtIicVt/j4gnsV8uL/0xL4S8d4?=
 =?us-ascii?Q?9Tmz3twds3hEIuBDiUMZlgmAoqM1oIH9Hi1QZzYNLupKvY1OMYu6iFEAdCh4?=
 =?us-ascii?Q?+OaTQCb7XrijJxxG6dmIRUBZV+xorPDn1WEXiYKQZ1CeX9CkayPWpOa8XWHK?=
 =?us-ascii?Q?WduxP/nFF8vA6ltisVQ4P3devBezuRselj6wEkQxgWMdHYVprv7fNSgvt/OW?=
 =?us-ascii?Q?1ekZZYO8R8uPHR3bH6C3OVsRO1nKTjNN5wljlEpl0BO/O9Mqcci9kbv+Zr4r?=
 =?us-ascii?Q?l7lAYrpYwBcH++dd1UY3pN2frsvZRgHcP0OJMLjnsmvcOAmeJPOL0j9Sp22+?=
 =?us-ascii?Q?+Z5cMOW+Oes2rE9EMHvz8IrWN09gJJz7Lx3+IvKR3mNBc2yM2nQXi1Si7MVS?=
 =?us-ascii?Q?NayDu7Egw0abznzneEYfzOQK1CR6324Fvlg1ct/zftzL3ZgcUsWxlVD3Tygv?=
 =?us-ascii?Q?50iSVnkITY1SN9ikWIKZ1ljX2yh6ayKVF1jQeJelvicEva5VUvFkzjpkDoYh?=
 =?us-ascii?Q?3RVO2un3C3QmQ37w7vtAYDiMkb94DJVlvgp76/L6yDCHi/JTxATjT1QrhhAo?=
 =?us-ascii?Q?bw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM8PR11MB5621.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 17788b6d-0bc8-46b6-71cd-08dac971a287
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Nov 2022 14:31:52.1150
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: or0ESeHYpv7yPJGHsLwCCvbTr5tHMfNKz5pDal4Uql88ruSZonmx8jFyCTA1bosv71nFlcqJj3jorxqxBufamhEpEAp+gl/w8Y1LqAMgiuQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4971
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of I=
van
> Vecera
> Sent: Tuesday, November 8, 2022 11:25 AM
> To: netdev@vger.kernel.org
> Cc: SlawomirX Laba <slawomirx.laba@intel.com>; Eric Dumazet
> <edumazet@google.com>; moderated list:INTEL ETHERNET DRIVERS <intel-
> wired-lan@lists.osuosl.org>; open list <linux-kernel@vger.kernel.org>;
> Piotrowski, Patryk <patryk.piotrowski@intel.com>; Jakub Kicinski
> <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; David S. Miller
> <davem@davemloft.net>; sassmann@redhat.com
> Subject: [Intel-wired-lan] [PATCH net] iavf: Do not restart Tx queues aft=
er reset
> task failure
>=20
> After commit aa626da947e9 ("iavf: Detach device during reset task") the d=
evice
> is detached during reset task and re-attached at its end.
> The problem occurs when reset task fails because Tx queues are restarted =
during
> device re-attach and this leads later to a crash.
>=20
> To resolve this issue properly close the net device in cause of failure i=
n reset task
> to avoid restarting of tx queues at the end.
> Also replace the hacky manipulation with IFF_UP flag by device close that=
 clears
> properly both IFF_UP and __LINK_STATE_START flags.
> In these case iavf_close() does not do anything because the adapter state=
 is
> already __IAVF_DOWN.
>=20
> Reproducer:
> 1) Run some Tx traffic (e.g. iperf3) over iavf interface
> 2) Set VF trusted / untrusted in loop
>=20
> [root@host ~]# cat repro.sh
> #!/bin/sh
>=20
> PF=3Denp65s0f0
> IF=3D${PF}v0
>=20
> ip link set up $IF
> ip addr add 192.168.0.2/24 dev $IF
> sleep 1
>=20
> iperf3 -c 192.168.0.1 -t 600 --logfile /dev/null & sleep 2
>=20
> while :; do
>         ip link set $PF vf 0 trust on
>         ip link set $PF vf 0 trust off
> done
> [root@host ~]# ./repro.sh
>=20
> Result:
> [ 2006.650969] iavf 0000:41:01.0: Failed to init adminq: -53 [ 2006.67566=
2] ice
> 0000:41:00.0: VF 0 is now trusted [ 2006.689997] iavf 0000:41:01.0: Reset=
 task
> did not complete, VF disabled [ 2006.696611] iavf 0000:41:01.0: failed to=
 allocate
> resources during reinit [ 2006.703209] ice 0000:41:00.0: VF 0 is now untr=
usted [
> 2006.737011] ice 0000:41:00.0: VF 0 is now trusted [ 2006.764536] ice
> 0000:41:00.0: VF 0 is now untrusted [ 2006.768919] BUG: kernel NULL point=
er
> dereference, address: 0000000000000b4a [ 2006.776358] #PF: supervisor rea=
d
> access in kernel mode [ 2006.781488] #PF: error_code(0x0000) - not-presen=
t
> page [ 2006.786620] PGD 0 P4D 0 [ 2006.789152] Oops: 0000 [#1] PREEMPT SM=
P
> NOPTI [ 2006.792903] ice 0000:41:00.0: VF 0 is now trusted [ 2006.793501]=
 CPU:
> 4 PID: 0 Comm: swapper/4 Kdump: loaded Not tainted 6.1.0-rc3+ #2 [
> 2006.805668] Hardware name: Abacus electric, s.r.o. - servis@abacus.cz Su=
per
> Server/H12SSW-iN, BIOS 2.4 04/13/2022 [ 2006.815915] RIP:
> 0010:iavf_xmit_frame_ring+0x96/0xf70 [iavf] [ 2006.821028] ice 0000:41:00=
.0:
> VF 0 is now untrusted [ 2006.821572] Code: 48 83 c1 04 48 c1 e1 04 48 01 =
f9 48
> 83 c0 10 6b 50 f8 55 c1 ea 14 45 8d 64 14 01 48 39 c8 75 eb 41 83 fc 07 0=
f 8f e9 08
> 00 00 <0f> b7 45 4a 0f b7 55 48 41 8d 74 24 05 31 c9 66 39 d0 0f 86 da 00=
 [
> 2006.845181] RSP: 0018:ffffb253004bc9e8 EFLAGS: 00010293 [ 2006.850397]
> RAX: ffff9d154de45b00 RBX: ffff9d15497d52e8 RCX: ffff9d154de45b00 [
> 2006.856327] ice 0000:41:00.0: VF 0 is now trusted [ 2006.857523] RDX:
> 0000000000000000 RSI: 00000000000005a8 RDI: ffff9d154de45ac0 [
> 2006.857525] RBP: 0000000000000b00 R08: ffff9d159cb010ac R09:
> 0000000000000001 [ 2006.857526] R10: ffff9d154de45940 R11:
> 0000000000000000 R12: 0000000000000002 [ 2006.883600] R13:
> ffff9d1770838dc0 R14: 0000000000000000 R15: ffffffffc07b8380 [ 2006.88584=
0]
> ice 0000:41:00.0: VF 0 is now untrusted [ 2006.890725] FS:
> 0000000000000000(0000) GS:ffff9d248e900000(0000) knlGS:0000000000000000
> [ 2006.890727] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> 2006.909419] CR2: 0000000000000b4a CR3: 0000000c39c10002 CR4:
> 0000000000770ee0 [ 2006.916543] PKRU: 55555554 [ 2006.918254] ice
> 0000:41:00.0: VF 0 is now trusted [ 2006.919248] Call Trace:
> [ 2006.919250]  <IRQ>
> [ 2006.919252]  dev_hard_start_xmit+0x9e/0x1f0 [ 2006.932587]
> sch_direct_xmit+0xa0/0x370 [ 2006.936424]  __dev_queue_xmit+0x7af/0xd00 [
> 2006.940429]  ip_finish_output2+0x26c/0x540 [ 2006.944519]
> ip_output+0x71/0x110 [ 2006.947831]  ? __ip_finish_output+0x2b0/0x2b0 [
> 2006.952180]  __ip_queue_xmit+0x16d/0x400 [ 2006.952721] ice 0000:41:00.0=
:
> VF 0 is now untrusted [ 2006.956098]  __tcp_transmit_skb+0xa96/0xbf0 [
> 2006.965148]  __tcp_retransmit_skb+0x174/0x860 [ 2006.969499]  ?
> cubictcp_cwnd_event+0x40/0x40 [ 2006.973769]
> tcp_retransmit_skb+0x14/0xb0 ...
>=20
> Fixes: aa626da947e9 ("iavf: Detach device during reset task")
> Cc: Jacob Keller <jacob.e.keller@intel.com>
> Cc: Patryk Piotrowski <patryk.piotrowski@intel.com>
> Cc: SlawomirX Laba <slawomirx.laba@intel.com>
> Signed-off-by: Ivan Vecera <ivecera@redhat.com>
> ---
>  drivers/net/ethernet/intel/iavf/iavf_main.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
>=20
> diff --git a/drivers/net/ethernet/intel/iavf/iavf_main.c
> b/drivers/net/ethernet/intel/iavf/iavf_main.c
> index 5abcd66e7c7a..b66f8fa1d83b 100644
> --- a/drivers/net/ethernet/intel/iavf/iavf_main.c
> +++ b/drivers/net/ethernet/intel/iavf/iavf_main.c
> @@ -2921,7 +2921,6 @@ static void iavf_disable_vf(struct iavf_adapter

Tested-by: Konrad Jankowski <konrad0.jankowski@intel.com>