Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 24 Sep 2021 11:10:04 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Dave Ertman <david.m.ertman@intel.com>
Cc:     davem@davemloft.net, kuba@kernel.org, yongxin.liu@windriver.com,
        shiraz.saleem@intel.com, anthony.l.nguyen@intel.com,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        jesse.brandeburg@intel.com, intel-wired-lan@lists.osuosl.org,
        leon@kernel.org, linux-rdma@vger.kernel.org
Subject: Re: [PATCH RESEND net] ice: Correctly deal with PFs that do not
 support RDMA
Message-ID: <20210924141004.GA1237721@nvidia.com>
References: <20210909151223.572918-1-david.m.ertman@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210909151223.572918-1-david.m.ertman@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cHj1hn4REdQ23Ua8yEGB0/ZkmssLqtXjTADonzM9invMSBP2DdTE3VUItGcC?=
 =?us-ascii?Q?f+tQyFZk7Oq5sT4wwkn1Oa8I+EuRj2iUyfCFhwAWoD40sm2HRrOaTvjVixkf?=
 =?us-ascii?Q?a6IL7H7MUMQNJuxfGpO+n18WiFjtI7ehIkTOClIciEFiZvwuC26YVnUN+nwz?=
 =?us-ascii?Q?+nslbb106KZ7Np7xIoDNzfKSmVQq4Kn4xUN9ffGAuQWeFZW552sLVIITjd6a?=
 =?us-ascii?Q?ddUbqobGbfqB9l9SM3bU5wtpCMEuNib50UF4l26TbmpFPeWUhKP/lys3liBg?=
 =?us-ascii?Q?gvWWzL0DezehLJzG2mmHw+Tofz1svNpnu05yhK0pidHzGOOzp+kxwEiXoDEC?=
 =?us-ascii?Q?pUPUZeZIbbCOTgfm3XZuZWkwryFYv4J8L6945cG2nTPx/xhmekeQAq084sWy?=
 =?us-ascii?Q?fLp92YiCdK0UeAOOV+XXzpQhT1jlratoGu/YTpmiOJcNvYm5qndL3teGcvZL?=
 =?us-ascii?Q?m5LlzmGSlMFIWSfSdBvM2+igpsW6e8wLXuT0Cdtr3AIK0PYpYa5IYfXfHXxx?=
 =?us-ascii?Q?DXIlKCkUDLtY9AQvPm4q71UUHmca9RmAbfe8wun6WfrCQtdeAsq3ZrQTrieD?=
 =?us-ascii?Q?xsxYlh/lKPCXCYgcEU30mN5oG9mFjb18zlWy3EJCkKHjC5Au52ECCv410EPM?=
 =?us-ascii?Q?qd2HF3QNDxL+uaMSmim/+MK9SClQ/Jm2H5VIhrAwf0Jv7vFZ8CR5FP4aO/sf?=
 =?us-ascii?Q?+8QjzCzQzOEK2n5gKcfM8hIKm3UR9Yet/iirZp1gYt7FikFG+D/QTcHOprcy?=
 =?us-ascii?Q?SUxWmkawDkiE4eIhwp8vXtVhpH0y+J9FrSY4soJNOH8ykv9K2aPQQrpRqoY3?=
 =?us-ascii?Q?JdYa5nsP7aARhwHxcJJvHZ+7MlpsLc/80JcAHnOiCplfmtGEGjhpVMsRjNv+?=
 =?us-ascii?Q?yO23Vv9wU8CEnYmiSbWpx66NxSPNkTaLyP+Jtm9Z9rtFIpCy7J37BDst6wE/?=
 =?us-ascii?Q?gJvTupIEI9bwNi9i+kdik01ZoiDUlgvvEkL1YfagiyqSUiT6z2ZGhZTibFsj?=
 =?us-ascii?Q?8yZAPB12vAgV7VdKmfyqAp7GlQzicQwS6XVHLFRzau92QzdmvEBfUXqKdjr+?=
 =?us-ascii?Q?Sp64MipqYr3Gr838HP+SRX9mH8exko2/adhM7dED18k9ceQzb4RARq38vpId?=
 =?us-ascii?Q?nkHzPqord0sOVITVbo7Ak/0WOnX2WfhowNUoX+Blb7RObuK5+WaYjaE5JSyL?=
 =?us-ascii?Q?pr5357C5ys1p7pJKmMkT2pYOYX/fJdooglwq1gc9BMf9BycSVwwXC10X261t?=
 =?us-ascii?Q?lOzn6aBhty+nDyN51mi1KLccYgpokJ83FHS05JD+W07FvRDIK9CBxcvgIGmF?=
 =?us-ascii?Q?al7XHP7tP0NLztDIi81N1FXv?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ff3869d1-c19a-4708-dc56-08d97f650270
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Sep 2021 14:10:06.1609
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: isEF8Hb1gvmEFAmZCc3iYzwGBVDKUo573WXUaiBefiQdjV6MK/CBHRyHlciUZoWt
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5112
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 09, 2021 at 08:12:23AM -0700, Dave Ertman wrote:
> There are two cases where the current PF does not support RDMA
> functionality.  The first is if the NVM loaded on the device is set
> to not support RDMA (common_caps.rdma is false).  The second is if
> the kernel bonding driver has included the current PF in an active
> link aggregate.
> 
> When the driver has determined that this PF does not support RDMA, then
> auxiliary devices should not be created on the auxiliary bus.  Without
> a device on the auxiliary bus, even if the irdma driver is present, there
> will be no RDMA activity attempted on this PF.
> 
> Currently, in the reset flow, an attempt to create auxiliary devices is
> performed without regard to the ability of the PF.  There needs to be a
> check in ice_aux_plug_dev (as the central point that creates auxiliary
> devices) to see if the PF is in a state to support the functionality.
> 
> When disabling and re-enabling RDMA due to the inclusion/removal of the PF
> in a link aggregate, we also need to set/clear the bit which controls
> auxiliary device creation so that a reset recovery in a link aggregate
> situation doesn't try to create auxiliary devices when it shouldn't.
> 
> Fixes: f9f5301e7e2d ("ice: Register auxiliary device to provide RDMA")
> Reported-by: Yongxin Liu <yongxin.liu@windriver.com>
> Signed-off-by: Dave Ertman <david.m.ertman@intel.com>
> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
>  drivers/net/ethernet/intel/ice/ice.h     | 2 ++
>  drivers/net/ethernet/intel/ice/ice_idc.c | 6 ++++++
>  2 files changed, 8 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h
> index eadcb9958346..3c4f08d20414 100644
> +++ b/drivers/net/ethernet/intel/ice/ice.h
> @@ -695,6 +695,7 @@ static inline void ice_set_rdma_cap(struct ice_pf *pf)
>  {
>  	if (pf->hw.func_caps.common_cap.rdma && pf->num_rdma_msix) {
>  		set_bit(ICE_FLAG_RDMA_ENA, pf->flags);
> +		set_bit(ICE_FLAG_AUX_ENA, pf->flags);
>  		ice_plug_aux_dev(pf);

I agree with Leon, there shouldn't be a flag for "aux en". aux is
enabled when a device on the aux bus is required. It should all be
rdma en, which already seems to have a bit.

Th only existing place that uses aux_ena immediately calls

		err = ice_init_rdma(pf);

So I'd just delete the whole thing and use rdma_ena. Frankly it looks
structured confusingly, the mlx implementation is better where this is
one function that synchronizes the aux bus with the current state of
the driver - adding/removing as required

Jason