Date:   Mon, 15 Nov 2021 11:09:31 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jiri Pirko <jiri@resnulli.us>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Jakub Kicinski <kuba@kernel.org>, Jiri Pirko <jiri@nvidia.com>,
        Ido Schimmel <idosch@idosch.org>,
        "David S . Miller" <davem@davemloft.net>,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
        edwin.peer@broadcom.com
Subject: Re: [PATCH net-next] devlink: Require devlink lock during device
 reload
Message-ID: <20211115150931.GA2386342@nvidia.com>
References: <20211109153335.GH1740502@nvidia.com>
 <20211109082042.31cf29c3@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
 <20211109182427.GJ1740502@nvidia.com>
 <YY0G90fJpu/OtF8L@nanopsycho>
 <YY0J8IOLQBBhok2M@unreal>
 <YY4aEFkVuqR+vauw@nanopsycho>
 <YZCqVig9GQi/o1iz@unreal>
 <YZJCdSy+wzqlwrE2@nanopsycho>
 <20211115125359.GM2105516@nvidia.com>
 <YZJx8raQt+FkKaeY@nanopsycho>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YZJx8raQt+FkKaeY@nanopsycho>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?WV3MCbCyJLBZTyLTOiDfRyISr3gy+gMML7uN1S9qus1RNWTXRbuNSaGssULn?=
 =?us-ascii?Q?DD4eQzVYZVRx1Yg+t5EJRNtTpTzYMgC7i/YEsBoJhCrkDuzYVpPMIA3jQyMh?=
 =?us-ascii?Q?OZEYwiCOY+e5an13g4u2jWusxTtXu/byZn9e6r6xTRE8GpHyoWgQaffPdx2K?=
 =?us-ascii?Q?ipZkqisM+LCNQLcL73LSPWff8UY/hLYRDUU9QCyEwoG7B22lQR7OqMAuHxp5?=
 =?us-ascii?Q?rvQVSYpz5XG7VlqFr/xIuIoieF8yyxC/xLBkCh1+8459js5V+6+sGLoxlyBe?=
 =?us-ascii?Q?cbRpdKTwUEQIqYoyQYuj1aGv5/RgSjE1SEnVchb8bX+I5uPjhFStavdpBdM0?=
 =?us-ascii?Q?klYMyjCQ5rZiTfmHgF1/jv0mau0+gBtGsrtHYcGgzH7c4glwHZTXuDVoZg+h?=
 =?us-ascii?Q?5vZG8i9AQh2s4OYS2hbDCEnlxCmonlOSip3q9p7T1JbZR3QQ3FOb6mhvEO2w?=
 =?us-ascii?Q?swe4AOU1JI2d+12DPgmtqSP6xijv976MkpiwmwyvI+dpIuc+Ox/G7UJCvCpu?=
 =?us-ascii?Q?qgimEAGy4yqw9rLKDKojf0AGIvwX18n38+4ejGZf34m9VmxswewUphpDv7py?=
 =?us-ascii?Q?SS8cGXZRwdrf/v+JA1Ez8pPzLTCFK8AvQUaqFflIX2oy6P0/fFcfQb5/h8vU?=
 =?us-ascii?Q?vt6vTud+n86jO01BSEBuEgIZj4yCAVlAO7ZRwUdcbniz8vg0tN+RDFXbmn/Q?=
 =?us-ascii?Q?pTq8ZjDnN5uT0/QgRlkqXcjVzCh9Z0YnrFD26GyJjUShuNa+5IvGSvX04y+Y?=
 =?us-ascii?Q?6glga/xpCFp6WwLvLbMdiksMywAPr85Pu+HM1Nx4DX5JcBLB8BWT53Renaq9?=
 =?us-ascii?Q?V3pVroJ2qq0wgc9TkPBC/A7zshW7MFbcuFEAxcT7I8SVnR+aXXlaBBqXpmFN?=
 =?us-ascii?Q?/knphwZlZbos4i1JeQh9n4K6SpbeVKWBjAsFDu9YAJegVzaCHmq3IGUekgvw?=
 =?us-ascii?Q?QBZVhswfaFuuKhB62lZPVQKIGaLWLo7SRkVm9p2wJWRWbPRqzR7M0z2tfY0+?=
 =?us-ascii?Q?iXhrj8spDU9/fBKFiUoTrJB7dNcym7pwJ1hRAG/IcGXnufOaeqlkfEFhZgSY?=
 =?us-ascii?Q?T19NKmB8uhtOLLeTvv+ti27j8bk8P8j5ViWlq+e3/sTp8HEgDqKc4cizpW6q?=
 =?us-ascii?Q?s/X0bMQywPv4AdyyeUbf8voH13laeFJEUFXvnt9s4K5CQLQhe4/ik7Tt50Cv?=
 =?us-ascii?Q?3YF/+5wtKZ14eamefFMvro1gb+TD5bC4S+HvYVFps1KaDdePnj61qULcJ9ah?=
 =?us-ascii?Q?91VjOYYT9FZaKjh8GCcfNIYmyfTtvENKUKlxqnzQcBJjBslDUycpmleh5GW+?=
 =?us-ascii?Q?m5hMkiCeOhL27weDzvagkD7DLif/6moy1W9TXRKoSAiTn1B54Rs1VWDJ+/IQ?=
 =?us-ascii?Q?sE2MgqA43lyppMBVkEhOLjgxzU3TwEtYASCIr5yJZGM0Sj/MOVRjjBjf2o0A?=
 =?us-ascii?Q?NqCJSCRoEPG/HqhWy3OBFxqEdYB80mHUeCrtA7di8xs/EWGog6mtGhQMfxiY?=
 =?us-ascii?Q?H9Wrtm+zsvEzLX2QiOnoHYhDeM4g6/KhYJKWmrA5CYLxClMxkeIlOnE0zgzj?=
 =?us-ascii?Q?l88Vw5SvmxtiPVbrO5Q=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 39ab99da-2465-48ac-acfe-08d9a849ee8c
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2021 15:09:33.9474
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3Lo4dDW5djThlW3GyZEuxDQaDWcWPbvtMvetz5+DcSJ9wka/mY94CUY4soQuZaMj
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5191
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Nov 15, 2021 at 03:42:58PM +0100, Jiri Pirko wrote:

> >Sorry, I don't agree that registering a net notifier in an aux device
> >probe function is non-standard or wrong.
> 
> Listening to events which happen in different namespaces and react to
> them is the non-standard behaviour which I refered to. If you would not
> need to do it, you could just use netns notofier which would solve your
> issue. You know it.

Huh?

It calls the bog standard

 register_netdevice_notifier() 

Like hundreds of other drivers do from their probe functions

Which does:

int register_netdevice_notifier(struct notifier_block *nb)
{
	struct net *net;
	int err;

	/* Close race with setup_net() and cleanup_net() */
	down_write(&pernet_ops_rwsem);

And deadlocks because devlink hols the pernet_ops_rwsem when it
triggers reload in some paths.

There is nothing wrong with a driver doing this standard pattern.

There is only one place in the entire kernel calling the per-ns
register_netdevice_notifier_dev_net() and it is burred inside another
part of mlx5 for some reason..

I believe Parav already looked at using that in rdma and it didn't
work for some reason I've forgotten. 

It is not that we care about events in different namespaces, it is
that rdma, like everything else, doesn't care about namespaces and
wants events from the netdev no matter where it is located.

Jason