Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 9 Nov 2021 11:33:35 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jakub Kicinski <kuba@kernel.org>, Jiri Pirko <jiri@nvidia.com>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Ido Schimmel <idosch@idosch.org>,
        Jiri Pirko <jiri@resnulli.us>,
        "David S . Miller" <davem@davemloft.net>,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
        edwin.peer@broadcom.com
Subject: Re: [PATCH net-next] devlink: Require devlink lock during device
 reload
Message-ID: <20211109153335.GH1740502@nvidia.com>
References: <YYgSzEHppKY3oYTb@unreal>
 <20211108080918.2214996c@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
 <YYlfI4UgpEsMt5QI@unreal>
 <20211108101646.0a4e5ca4@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
 <YYlrZZTdJKhha0FF@unreal>
 <20211108104608.378c106e@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
 <YYmBbJ5++iO4MOo7@unreal>
 <20211108153126.1f3a8fe8@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
 <20211109144358.GA1824154@nvidia.com>
 <20211109070702.17364ec7@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20211109070702.17364ec7@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xi+W98VTCTxcWjtelFSefqu6A9l1lF6XXvzym+x25vNmgeXpGlUlmecCR3fQ?=
 =?us-ascii?Q?jrk5kQ9j4Sw1FtVLiHZBaF5YeJiSm8bT3xzGV4UP09e3S6M8CNpbCpjd9Wph?=
 =?us-ascii?Q?nKNbRGMsMyR0q6vnmH0oNx3UbNBnwqUHOrmll5RNqvf8xvpZpNw+ISt15ocV?=
 =?us-ascii?Q?JW9GtduNWi/lPAO8clNWKDB3BPtczf02uOfyOosAHq/aez34qmXS1xgYz5Mg?=
 =?us-ascii?Q?lvq4TzJmXZhEk3nenJYKz7XJzEYr4L0mvlRcMByKqlw3kKHTdEXLPrZyDk69?=
 =?us-ascii?Q?nmfATQWYLGmxDP1N4NUCHDseiKiwPQv7D5zJj2V3S+1dTqzOcy8QEw3kD7TU?=
 =?us-ascii?Q?m4BF+0+IaMHvEkDeU8yqLJsKn607aMjG2gyCEREL9VBd57X4z21RFKf4cJO/?=
 =?us-ascii?Q?9VjHuRftetfS6xxCVbY7CWDfXLoVEN72/XJ6mMSUa1cpHESq0olmShExhBc8?=
 =?us-ascii?Q?nNIQ61zl3CqXekBjAHd5SQHHq1I9PG6OOnW70kWvN0D31R0NQvH//5dzJoul?=
 =?us-ascii?Q?fiiwyg0wvqLd96AkGelCFnj1x3uml6/U1JaevLhfYv4GSEsiD+YeqwS1m8qK?=
 =?us-ascii?Q?ufWlSiP7drY+ubn5IqPMkg14DqDksCwJiWhAEkU38htLBRt/LxhtSlXRB6D9?=
 =?us-ascii?Q?Fx7reU1Y1GU4Z5d4ENcDeQlOQn/QXCNhHY/mx6LmEhfuOefd6OtVm4sT3NAg?=
 =?us-ascii?Q?u2OxcSNJpPTojD0VWaxjFZ5XSjoDh/Ia7kE/3EwqqB4t7EY4jsEebKOk80Tu?=
 =?us-ascii?Q?mBBGH5ne0dUXlgSPXLkoIxQ+3B3yGEYdTpYuqSWgmDDNYoaMJC9X+54Lf2yE?=
 =?us-ascii?Q?tgqqIBkf7XswaNTlQjS1p3mQhQ3O8r/gjAcjGNVHURfQ+2L7DcL2lrolWC0c?=
 =?us-ascii?Q?ul+mb0aLUfhr1uBDkNo/TpLB8FTl+qkcUTFNVNKC6ziaIt5TKvxsuVn6hcLq?=
 =?us-ascii?Q?LXTQG0bMlbKJ9eSZSiX/QPWJ2ExqKK6q8lx6XdyWtwgfVwH/R3+BBfUZMpTv?=
 =?us-ascii?Q?bzojE713QKoLKaC/YaPUK4h+5nmYartrfJ6IMhTDPaE6WpYMLJUp80LXWa5X?=
 =?us-ascii?Q?NfXrtqCDgJMsscZ26Rq/0gXomouzdf6w79TwCjo3CdK1Ac1fCun9DMI6SRzA?=
 =?us-ascii?Q?7CBaZO4rwOnGbx/XKD7Qj+gVGZJRbxKtOOMcw+EIGbCZ2edqEX9kpCKS6y1a?=
 =?us-ascii?Q?jDXqmmOCHJ/aqVJKJ3yGTILHB6T+hPKhy94iC/j4h7+rBupZ6TVSFhjGQTeU?=
 =?us-ascii?Q?6BvDJvY8erTQdYQe3O3w/uKmzyYYELB1TAT1UyxeHnTLTZTT+nAHcxh3OZI4?=
 =?us-ascii?Q?YtYvX41vTIl+yh+PoDPUsuWuVOCvlXb97l95Y2pCyRVs2TRUqKQQQfOZnUDe?=
 =?us-ascii?Q?kqa4GfLGCQkCogZQ2iKAU+udehD6UMWCp+HP8Cy8JLRVkwUIrgh/fhbjtJZw?=
 =?us-ascii?Q?nYzihbr1OL9y+qC8s4J4eMyrwMCvBHsqTxx8apm9VTKRZ3y5qN2wDk+s83/v?=
 =?us-ascii?Q?NAEI66BsaaPE8A/nMzgHEb2pUa+ErGLIMw3AU5xiVlS1fmXRnT+IZU/rq46R?=
 =?us-ascii?Q?9YweaAkBftuIoDCYHjs=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3237987e-64f9-4a61-2c40-08d9a3964c13
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2021 15:33:36.7955
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KK74hnxeoXoFqT/UjT4T8+VFoBdSF+8iGgMI7J7dHn30DtWE2HSf/Dp3Q15cUiXV
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5351
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 09, 2021 at 07:07:02AM -0800, Jakub Kicinski wrote:
> On Tue, 9 Nov 2021 10:43:58 -0400 Jason Gunthorpe wrote:
> > This becomes all entangled in the aux device stuff we did before.
> 
> So entangled in fact that neither of you is willing to elucidate 
> the exact need ;)

I haven't been paying attention to this thread, Leon just asked me to
elaborate on why roce is in these traces.

What I know is mlx5 testing has shown an alarming number of crashers
and bugs connected to devlink and Leon has been grinding them
away. mlx5 is quite heavily invested in devlink and mlx5 really needs
it to work robustly.

> The main use case for reload for netns is placing a VF in a namespace,
> for a container to use. Is that right? I've not seen use cases
> requiring the PF to be moved, are there any?

Sure, it can be useful. I wouldn't call it essential, but it is there.
 
> > I once sketched out fixing this by removing the need to hold the
> > per_net_rwsem just for list iteration, which in turn avoids holding it
> > over the devlink reload paths. It seemed like a reasonable step toward
> > finer grained locking.
> 
> Seems to me the locking is just a symptom.

My fear is this reload during net ns destruction is devlink uAPI now
and, yes it may be only a symptom, but the root cause may be unfixable
uAPI constraints. Jiri, what do you think?

Speaking generally, I greatly prefer devlink move toward a design where
the aux bus operations that must be connected with devlink reload are
not invoked under any global locks at all. Not per_net_rwsem, rtnl, or
devlink BKLs.

We know holding global locks in open-ended contexts, like driver core
device_register/unregister, is an dangerous pattern.

Concretely, now that we are pushing virtualization use cases into
using devlink as a control plane mlx5 has pod/VM launch issuing
devlink steps. Some of this stuff is necessarily slow, like attaching
a roce aux driver takes a while. Holding a BKL while doing that means
all VM launches are serialized.

IMHO these needs demand a fine grained locking approach, not a BKL
design.

Jason