Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Keller, Jacob E" <jacob.e.keller@intel.com>
To:     ivecera <ivecera@redhat.com>,
        "Fijalkowski, Maciej" <maciej.fijalkowski@intel.com>
CC:     "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "moderated list:INTEL ETHERNET DRIVERS" 
        <intel-wired-lan@lists.osuosl.org>, mschmidt <mschmidt@redhat.com>,
        Brett Creeley <brett.creeley@intel.com>,
        open list <linux-kernel@vger.kernel.org>,
        poros <poros@redhat.com>, Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        "David S. Miller" <davem@davemloft.net>
Subject: RE: [Intel-wired-lan] [PATCH net] ice: Fix incorrect locking in
 ice_vc_process_vf_msg()
Thread-Topic: [Intel-wired-lan] [PATCH net] ice: Fix incorrect locking in
 ice_vc_process_vf_msg()
Thread-Index: AQHYRO057BJtvqZxeEyUMOJlKAIrPqzZeKaAgAArCwCAAEY+YA==
Date:   Thu, 31 Mar 2022 20:02:43 +0000
Message-ID: <CO1PR11MB50891AEAC5604FD63BEB5C76D6E19@CO1PR11MB5089.namprd11.prod.outlook.com>
References: <20220331105005.2580771-1-ivecera@redhat.com>
        <YkWpNVXYEBo/u3dm@boxer> <20220331174832.68e17c4a@ceranb>
In-Reply-To: <20220331174832.68e17c4a@ceranb>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ZVf8458fyFY4rQqd6TKf8OCA48+YjLWNLvaqK/ms6psB3vXCsY+DuFWxjpM7?=
 =?us-ascii?Q?cSWjGNE6mYmS/+lSt0IvnPlpQik7851fxs+L8o1OcZDyTjPdHu5JqBYq3O2E?=
 =?us-ascii?Q?uCeun3OD1oAB1PObDkiHxm0bJDdu0islJGuBmjwJPPdZNsCthMYojV63fk71?=
 =?us-ascii?Q?mguiNW6r0D3z3AEBmVxnsNdPIjMuD5BP8ygmcEITfcUh1yv/FIgwFG2V2L+4?=
 =?us-ascii?Q?X2VWyA3FG0cO/FF5w/+sbGynH3Ia8d0b2spObjS0GFaHNu0401jwEvEynfSd?=
 =?us-ascii?Q?NpvmafelaQPv5LbWlDcExsDD79gxJc1c/h+VS7gnnyG/5PCeQOvx4J7vLd54?=
 =?us-ascii?Q?uBUNSi1u9pAN33ej/qX+j82SpEMwnSC9967jZgMUaH9gvjPErRZ5r+HF6v+M?=
 =?us-ascii?Q?bYWRtgog0cWvqF9t37D64eQAFwiiUfzTL7ickUqnDWvXnJRcTmctZa8sZx6O?=
 =?us-ascii?Q?MeApdbC9CpWjs7qnsDS3+qLGEa+cU/Ltybze+9X7GT6iE6goj0yTYjpiALSZ?=
 =?us-ascii?Q?ohujNnEu1l2WllsqMLypdVzapUsqa5/xiYXXzDUsmKH3BCLjtPmpoWnBBEl2?=
 =?us-ascii?Q?OnGuExdo2pl/ujX94e9blb7dxf2LdgmpZ1QykVc1EzD8BoN/GvKK2GBvUvSU?=
 =?us-ascii?Q?2GLbqFksJifIWjhpZ06UVKT2c7lOBvxRnu1KTy15ixUkMlohImORp409EzX5?=
 =?us-ascii?Q?d3c/lrD2IWO01On08y9RqV+Z70YaoQHCPkSx5N/zCfrtfv0FQ7DC5C8AkcIl?=
 =?us-ascii?Q?RKYGE0rAdtDHBOGuBZoNtTlPAqzy1H2hs3OhDnHSGrngY/XnS0jLMEjHHHwQ?=
 =?us-ascii?Q?gmnRYL+C6V+ffH52lBPe+XK8KBkR2dyUUy9b79wqFTOU3ktHQdX2N0RWk4Ho?=
 =?us-ascii?Q?6h4+DMchZjgSvqCzbrrTpRg/mEwpsesfUaO1+BbCmm/6PaaSIWct/J667yyD?=
 =?us-ascii?Q?oAzAYjXUdbiuBVA+hrdsIBrvRF1VXFIbQ9le1sgU16GN+HAg9DqI7jQOOE/w?=
 =?us-ascii?Q?y53lPiwopFRUPkNg1j2N5cZbqc/30PiUbM159C9ymJB8dGP8CmhOBL1uzib3?=
 =?us-ascii?Q?myPHcfXlbpgUFH3poyum8DOdKEkTCWuu7D5lt8XH7cLuHP35lcsN3hA7lDol?=
 =?us-ascii?Q?OqmFXRc2GYtJFGb22a47lxsdm0Egyer1Pq1Kjl6s2G70AJHV7pzBY3jAFLtb?=
 =?us-ascii?Q?gJd5FLrokXJjQBymfZ1Z5tWQFWy8M+KQak3ZTDMbNFbwghBJo03xota9Malu?=
 =?us-ascii?Q?SRviJU784LTXHlNc/kyw2Nb1cpqfGalsec7IlAYIOvzAAHP9l6rjCaZtwn3z?=
 =?us-ascii?Q?P6JA5SIgXI/FQzod/RD0PKjvI8kwmJ3ZZrJbnOK5CjlmKEJBRBT8rEInWPrq?=
 =?us-ascii?Q?aBserIJsnLEw2bd0WvoRQ22QtOX+DSvAhwTm/1fa1ZyiQ0HvGsFNCC3pOzCL?=
 =?us-ascii?Q?Cdal81r89IRPkE4f8BtFHisHcf5jTWTAcb3xVKmPRKgZGldZDe0P88+EOTLq?=
 =?us-ascii?Q?9fxLWIfi9po21+9hQhsmxjTJnyk7X5agivDHZPfUf1Hi4h56OXMGjGTW1+kh?=
 =?us-ascii?Q?JQpgHi88Ro+F/kZboP9F7xYu5fXe89MpuF28PUsWFE8L+oQWd+/Yg1XQLO/M?=
 =?us-ascii?Q?AirvyX1fUZ9UIKgWzMyvWuDczMrXOr6QHwhflNq0y7/49FIxqPaybAwnvIel?=
 =?us-ascii?Q?BYbuJ03X+hGOlPyEmGg/QHNkLkWnandN+hZDYA4iMjGMEbVTPEX6Og3z1ikO?=
 =?us-ascii?Q?uat7fgvSDg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3622b626-6dbc-46a4-76db-08da13516b40
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Mar 2022 20:02:43.7606
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: VO3NHVLpPqaVyTM3MZDKoxoStLZURQsSCnZuxM/NBTtK2ioJHJqZ5sobhOPAtQ5tIwxUoOBJDBKxt786ukC8bC7AHMFTZKVu1tFpYKATGrI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1840
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Ivan Vecera <ivecera@redhat.com>
> Sent: Thursday, March 31, 2022 8:49 AM
> To: Fijalkowski, Maciej <maciej.fijalkowski@intel.com>
> Cc: netdev@vger.kernel.org; moderated list:INTEL ETHERNET DRIVERS <intel-
> wired-lan@lists.osuosl.org>; mschmidt <mschmidt@redhat.com>; Brett Creele=
y
> <brett.creeley@intel.com>; open list <linux-kernel@vger.kernel.org>; poro=
s
> <poros@redhat.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; David S. Miller <davem@davemloft.net>
> Subject: Re: [Intel-wired-lan] [PATCH net] ice: Fix incorrect locking in
> ice_vc_process_vf_msg()
>=20
> On Thu, 31 Mar 2022 15:14:29 +0200
> Maciej Fijalkowski <maciej.fijalkowski@intel.com> wrote:
>=20
> > On Thu, Mar 31, 2022 at 12:50:04PM +0200, Ivan Vecera wrote:
> > > Usage of mutex_trylock() in ice_vc_process_vf_msg() is incorrect
> > > because message sent from VF is ignored and never processed.
> > >
> > > Use mutex_lock() instead to fix the issue. It is safe because this
> >
> > We need to know what is *the* issue in the first place.
> > Could you please provide more context what is being fixed to the reader=
s
> > that don't have an access to bugzilla?
> >
> > Specifically, what is the case that ignoring a particular message when
> > mutex is already held is a broken behavior?
>=20
> Reproducer:
>=20
> <code>
> #!/bin/sh
>=20
> set -xe
>=20
> PF=3D"ens7f0"
> VF=3D"${PF}v0"
>=20
> echo 1 > /sys/class/net/${PF}/device/sriov_numvfs
> sleep 2
>=20
> ip link set ${VF} up
> ip addr add 172.30.29.11/24 dev ${VF}
>=20
> while true; do
>=20
> # Set VF to be trusted
> ip link set ${PF} vf 0 trust on
>=20
> # Ping server again
> ping -c5 172.30.29.2 || {
>         echo Ping failed
>         ip link show dev ${VF} # <- No carrier here
>         break
> }
>=20
> ip link set ${PF} vf 0 trust off
> sleep 1
>=20
> done
>=20
> echo 0 > /sys/class/net/${PF}/device/sriov_numvfs
> </code>
>=20
> <sample>
> [root@wsfd-advnetlab150 ~]# uname -r
> 5.17.0+ # Current net.git HEAD
> [root@wsfd-advnetlab150 ~]# ./repro_simple.sh
> + PF=3Dens7f0
> + VF=3Dens7f0v0
> + echo 1
> + sleep 2
> + ip link set ens7f0v0 up
> + ip addr add 172.30.29.11/24 dev ens7f0v0
> + true
> + ip link set ens7f0 vf 0 trust on
> + ping -c5 172.30.29.2
> PING 172.30.29.2 (172.30.29.2) 56(84) bytes of data.
> 64 bytes from 172.30.29.2: icmp_seq=3D2 ttl=3D64 time=3D0.820 ms
> 64 bytes from 172.30.29.2: icmp_seq=3D3 ttl=3D64 time=3D0.142 ms
> 64 bytes from 172.30.29.2: icmp_seq=3D4 ttl=3D64 time=3D0.128 ms
> 64 bytes from 172.30.29.2: icmp_seq=3D5 ttl=3D64 time=3D0.129 ms
>=20
> --- 172.30.29.2 ping statistics ---
> 5 packets transmitted, 4 received, 20% packet loss, time 4110ms
> rtt min/avg/max/mdev =3D 0.128/0.304/0.820/0.298 ms
> + ip link set ens7f0 vf 0 trust off
> + sleep 1
> + true
> + ip link set ens7f0 vf 0 trust on
> + ping -c5 172.30.29.2
> PING 172.30.29.2 (172.30.29.2) 56(84) bytes of data.
> From 172.30.29.11 icmp_seq=3D1 Destination Host Unreachable
> From 172.30.29.11 icmp_seq=3D2 Destination Host Unreachable
> From 172.30.29.11 icmp_seq=3D3 Destination Host Unreachable
>=20
> --- 172.30.29.2 ping statistics ---
> 5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4125=
ms
> pipe 3
> + echo Ping failed
> Ping failed
> + ip link show dev ens7f0v0
> 20: ens7f0v0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
> state DOWN mode DEFAULT group default qlen 1000
>     link/ether de:69:e3:a5:68:b6 brd ff:ff:ff:ff:ff:ff
>     altname enp202s0f0v0
> + break
> + echo 0
>=20
> [root@wsfd-advnetlab150 ~]# dmesg | tail -8
> [  220.265891] iavf 0000:ca:01.0: Reset indication received from the PF
> [  220.272250] iavf 0000:ca:01.0: Scheduling reset task
> [  220.277217] iavf 0000:ca:01.0: Hardware reset detected
> [  220.292854] ice 0000:ca:00.0: VF 0 is now trusted
> [  220.295027] ice 0000:ca:00.0: VF 0 is being configured in another cont=
ext that
> will trigger a VFR, so there is no need to handle this message
> [  234.445819] iavf 0000:ca:01.0: PF returned error -64 (IAVF_NOT_SUPPORT=
ED)
> to our request 9
> [  234.466827] iavf 0000:ca:01.0: Failed to delete MAC filter, error
> IAVF_NOT_SUPPORTED
> [  234.474574] iavf 0000:ca:01.0: Remove device
> </sample>
>=20
> User set VF to be trusted so .ndo_set_vf_trust (ice_set_vf_trust) is call=
ed.
> Function ice_set_vf_trust() takes vf->cfg_lock and calls ice_vc_reset_vf(=
) that
> sends message to iavf that initiates reset task. During this reset task i=
avf sends
> config messages to ice. These messages are handled in ice_service_task() =
context
> via ice_clean_adminq_subtask() -> __ice_clean_ctrlq() ->
> ice_vc_process_vf_msg().

Right. Because the reset isn't finished in the PF by the time that the call=
er starts sending messages back.

I also think that this could be buggy if cfg_lock is held elsewhere too (th=
ough reset is the most likely problem).

Especially since the recent changes we did in ice to hold cfg_lock in more =
places to protect against concurrently configuring VFs. I think I agree wit=
h Ivans change (though perhaps we should re-test some cases for why we made=
 this a try lock originally).

The only other concern was mentioned in a different message by Brett. Perha=
ps we also want to cancel any outstanding messages from the VF when we star=
t a reset (since we're going to reset the VF and we don't really want to pr=
ocess any of its messages that were issued before the reset).

Thanks,
Jake

>=20
> Function ice_vc_process_vf_msg() tries to take vf->cfg_lock but this can =
be locked
> from ice_set_vf_trust() yet (as in sample above). The lock attempt failed=
 so the
> function
> returns, message is not processed.
>=20
> Thanks,
> Ivan