Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <2a630a76-bcfb-d08d-619d-eafa6a7b1025@intel.com>
Date:   Mon, 14 Nov 2022 16:49:15 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.1
Subject: Re: [PATCH net] iavf: Fix a crash during reset task
To:     Stefan Assmann <sassmann@redhat.com>,
        Ivan Vecera <ivecera@redhat.com>
CC:     <netdev@vger.kernel.org>, Jacob Keller <jacob.e.keller@intel.com>,
        "Patryk Piotrowski" <patryk.piotrowski@intel.com>,
        SlawomirX Laba <slawomirx.laba@intel.com>,
        Jesse Brandeburg <jesse.brandeburg@intel.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        "moderated list:INTEL ETHERNET DRIVERS" 
        <intel-wired-lan@lists.osuosl.org>,
        open list <linux-kernel@vger.kernel.org>
References: <20221108093534.1957820-1-ivecera@redhat.com>
 <20221108105343.vjczwdxcsxhfghk7@p1>
Content-Language: en-US
From:   Tony Nguyen <anthony.l.nguyen@intel.com>
In-Reply-To: <20221108105343.vjczwdxcsxhfghk7@p1>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Mi96UTZla05jcFF1VXFQQlgxNVZ1dVdYNkR1VVRCM3k1d0c5SGMwVU9LZVN0?=
 =?utf-8?B?M2ZoQWgrQjJ1blY4MXRJbzIxOS9zbTZKaW1XN29LK3I2V1NOT2htbWphRHQw?=
 =?utf-8?B?UTI0Tk9zWGFRSDV5VitIUEZpSFgyblMwdFBtWWY2NVFsYzJyUFZVdms3cnMr?=
 =?utf-8?B?eWplTTc1UDBkaVFYMmVQR01IL3JEdXBhUDFxL2p3eG9UR0VMbmRUVjVNdWN6?=
 =?utf-8?B?YmNBbWhwbUZOK2RqUXRyb2JmdDhTNlp0ZjV3R1hndTk4TjE5dm5zdjgzci9x?=
 =?utf-8?B?UXIwRXdsVkYzV2tvaS9uUDFvaGtidW9vMXJMS3FUelhUYytuN2g3dHdaV2hI?=
 =?utf-8?B?cmlmNm9oaEp3UU1ZSWUrLzNIWHduUlQxUlVJRTRzSzMwbUJ0TGpqeUJacUZM?=
 =?utf-8?B?T2I1UEZEN2t3cmpFVVNybXZ4U2FweUNxRzVaVkhqSURXT3MyUldFU3A2Z3Zs?=
 =?utf-8?B?T1M0QS9BZ0J5M3R2R1AyY1NJa050b2VMb0ZuV2xWZXRVVWZpSHR4bFRwYWZK?=
 =?utf-8?B?TTRzZWhNVm5FR0ptNlovNE1FOERSTUswRk9pVDhXdWFKaFVCU210QWlRRUY4?=
 =?utf-8?B?NGttRG93WDBhQU5teWg4NTUwK2R6UE9ydkhHc3o1Lzk1MmpmZEN3ZStxSHhZ?=
 =?utf-8?B?aldubWNwYmFYUUIyaTdyZWtMNERXZDJEc2hESnBiSzlRQVZ0VSs0TnJsZDhy?=
 =?utf-8?B?VWtOcGQzZHo0SWJGN1ZRQkRvWWRuUDFLU2kzT0FLQWttS0ZiS2l0MDE4TWFy?=
 =?utf-8?B?N01lZGVhZE1XVStTYnNDeWN5NjZnTFFxUTBNQVhKcTByNzhBTzZBT0RFM0Jy?=
 =?utf-8?B?NWZqMldmdlJhcjBaYlRmVEpxcFNqK3VCTklabHNNVThTd3FaZnk3Z3IwK3Vh?=
 =?utf-8?B?c0RESGlBQ29PbTZ2UXlaOG9WdzFIZ0FqMVRZbTBHUTZBbGNVMVArYXNGeWpa?=
 =?utf-8?B?WkQzcUl4V3ExaWtSV2h2RldKOEd1b3R0UEdsMFg2UWhPUU1wVEl4Vm1KQ2hi?=
 =?utf-8?B?R1RjL0ttZTVjMjBIRXNkSDJqNnZhLzQ4RnhQd01xOGRQK21LNFpBSk9ZSUha?=
 =?utf-8?B?bThNSFZwL25hZHIzK2x3dTFiVDVXMW9QTjBITWJyT2dMQWhFZjRua3RiVmM5?=
 =?utf-8?B?QmgzY09VajJSMVZKam5UVTJ3RUNMUWgxZ2dHdUJnVWdlTlVramptM0Q5czdw?=
 =?utf-8?B?UlUvbFZNbTlPalhPZGdhaTFDSE5wYml4NlRRTnRManF2cUV6UGNMOGdxSkNX?=
 =?utf-8?B?ZTJFVFVtRmNPYkdWRjFGSHRJM0orZGxKNHJwTldWbmZycTBBdXJVSHJQZUYv?=
 =?utf-8?B?Sk1qTjNhU0ZURUVJdW84RElsQ2QweGY5VlJPVktEVGpEMzVxY01YMEZveDFa?=
 =?utf-8?B?eE1HOE9YZzEzMlFoY0FiYnRQMW4yT3JjUFNXUWtsYjl2Wmx5dFowZ1ZoMUJI?=
 =?utf-8?B?TE1mZmJPZFF5VU5EVCtwL0Nod3ZoK21ad1daSUI0SnlHQldCa0R3WlI4NzZp?=
 =?utf-8?B?Y09pMEJpQ1Z1T0dGR1laS2RpV1M1bzY3cTRzaXlnUWJaTlJ3US9lYUljRTV1?=
 =?utf-8?B?R2pLOGZxS0s3RWxqbnJPUklGRW9SS0tMVVVrOFUwS2xGcUpDeEtrSFR1OGly?=
 =?utf-8?B?VWRLL1NNcTJiTkFzaHdDRHU3QStsbUoxU3g4R3hROHV4R0EvV0FTVDhQayt0?=
 =?utf-8?B?cGNyM2EwTWRKb1hEVStjRzlGditkUWZ3bWJZWlZLaWhvL0t3QzB4dUFCb0R0?=
 =?utf-8?B?YWNoenUxamJtMlRUcGJPci9BUlYyVnp3L3p0anppWjFYa1d1MFA3NE8vK3g2?=
 =?utf-8?B?RVovenpYTmFOVnFjbXlxZFRYQzlza1Z5c1ZYNUJaZ2Z4RUR2bkdHbnpiQUJi?=
 =?utf-8?B?RjQwa010TVh5TnQ5TWVpcExaNmdzSzZEaXpDekR4c0JmekUxOStiQVROQjBI?=
 =?utf-8?B?Q2s4YXZjbjg0eW1VeVRqY3UwV00xd3JCdDdRZEo4SkdyRmFqcGEyQlNyaFhs?=
 =?utf-8?B?T0FvVUJxM09CV09qNDh1VE9EZHFrTldaSVZhV1NoVDc2YWpPZkZveXdhMmcx?=
 =?utf-8?B?T0xSSU5OTnpjT3FscURKTkV6UXVCdjY1b2tqNUpvc3QrVFFhYUxCSVZyR25E?=
 =?utf-8?B?cUY2QWJndjJUWmN4SGVuR1BDSDhvV0hQcDRWMVVVWkRDM2lIOUxmRURsQjJm?=
 =?utf-8?B?K0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5851d5ba-895d-4604-888d-08dac6a33a45
X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3229.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2022 00:49:18.7825
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dk9BX3iowH1o1NbKghMw6vijn8PboMfaziEvvNJHnY0FTDofG1qiz/ignAeVZtnGTAx2i+XtNqGlWiUoUvfiVgOEmo5pxOgBls5mGDqmukY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6519
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/8/2022 2:53 AM, Stefan Assmann wrote:
> On 2022-11-08 10:35, Ivan Vecera wrote:
>> Recent commit aa626da947e9 ("iavf: Detach device during reset task")
>> removed netif_tx_stop_all_queues() with an assumption that Tx queues
>> are already stopped by netif_device_detach() in the beginning of
>> reset task. This assumption is incorrect because during reset
>> task a potential link event can start Tx queues again.
>> Revert this change to fix this issue.
>>
>> Reproducer:
>> 1. Run some Tx traffic (e.g. iperf3) over iavf interface
>> 2. Switch MTU of this interface in a loop
>>
>> [root@host ~]# cat repro.sh
>> #!/bin/sh
>>
>> IF=enp2s0f0v0
>>
>> iperf3 -c 192.168.0.1 -t 600 --logfile /dev/null &
>> sleep 2
>>
>> while :; do
>>          for i in 1280 1500 2000 900 ; do
>>                  ip link set $IF mtu $i
>>                  sleep 2
>>          done
>> done
> 
> With this patch applied iavf doesn't crash anymore but after a few
> cycles with the reproducer tx timeouts are observed.
> 
> [   47.551151] iavf 0000:00:09.0 eth0: NIC Link is Up Speed is 10 Gbps Full Duplex
> [   54.035902] ------------[ cut here ]------------
> [   54.037397] NETDEV WATCHDOG: eth0 (iavf): transmit queue 3 timed out
> [   54.039264] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:526 dev_watchdog+0x20f/0x250
> [   54.041524] Modules linked in: 8021q intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass rapl pcspkr drm ramoops reed_solomon crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic pata_acpi ghash_clmulni_intel ata_piix aesni_intel crypto_simd iavf libata be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> [   54.049723] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.1.0-rc2+ #90
> [   54.051049] Hardware name: Red Hat KVM, BIOS 1.15.0-2.module+el8.6.0+14757+c25ee005 04/01/2014
> [   54.052898] RIP: 0010:dev_watchdog+0x20f/0x250
> [   54.053907] Code: 00 e9 4d ff ff ff 48 89 df c6 05 92 24 96 01 01 e8 c6 f2 f8 ff 44 89 e9 48 89 de 48 c7 c7 28 7f f6 a0 48 89 c2 e8 6e 65 23 00 <0f> 0b e9 2f ff ff ff e8 25 06 2a 00 85 c0 74 b5 80 3d 74 1b 96 01
> [   54.057282] RSP: 0018:ffffaf56c00e0e80 EFLAGS: 00010282
> [   54.058164] RAX: 0000000000000000 RBX: ffff993ed95b8000 RCX: 0000000000000103
> [   54.059345] RDX: 0000000000000103 RSI: 00000000000000f6 RDI: 00000000ffffffff
> [   54.060473] RBP: ffff993ed95b8508 R08: 0000000000000000 R09: c0000000fff7ffff
> [   54.061558] R10: 0000000000000001 R11: ffffaf56c00e0d18 R12: ffff993ed95b8420
> [   54.062640] R13: 0000000000000003 R14: ffff993ed95b8508 R15: ffff993ef74a06c0
> [   54.063681] FS:  0000000000000000(0000) GS:ffff993ef7480000(0000) knlGS:0000000000000000
> [   54.064867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   54.065654] CR2: 00007f42309e1280 CR3: 0000000107f6a003 CR4: 0000000000170ee0
> [   54.066612] Call Trace:
> [   54.066985]  <IRQ>
> [   54.067265]  ? mq_change_real_num_tx+0xd0/0xd0
> [   54.067844]  call_timer_fn+0xa1/0x2c0
> [   54.068330]  ? mq_change_real_num_tx+0xd0/0xd0
> [   54.068916]  run_timer_softirq+0x527/0x550
> [   54.069447]  ? lock_is_held_type+0xd8/0x130
> [   54.069998]  __do_softirq+0xc3/0x481
> [   54.070469]  irq_exit_rcu+0xe4/0x120
> [   54.070963]  sysvec_apic_timer_interrupt+0x9e/0xc0
> [   54.071604]  </IRQ>
> [   54.071909]  <TASK>
> [   54.072223]  asm_sysvec_apic_timer_interrupt+0x16/0x20
> [   54.072942] RIP: 0010:default_idle+0x10/0x20
> [   54.073533] Code: 89 df 31 f6 5b 5d e9 ff 1c a5 ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 eb 07 0f 00 2d f2 2a 42 00 fb f4 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 65
> 
> This only occurs when the device is detached and reattached during reset.

Hi Ivan,

Was there going to be an update to the patch to resolve this? If not, 
I'll take what there is now.

Thanks,
Tony