Date:   Mon, 15 Nov 2021 07:22:06 -0800
From:   Jakub Kicinski <kuba@kernel.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Jiri Pirko <jiri@resnulli.us>, Leon Romanovsky <leon@kernel.org>,
        Jiri Pirko <jiri@nvidia.com>, Ido Schimmel <idosch@idosch.org>,
        "David S . Miller" <davem@davemloft.net>,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
        edwin.peer@broadcom.com
Subject: Re: [PATCH net-next] devlink: Require devlink lock during device
 reload
Message-ID: <20211115072206.72435d60@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
In-Reply-To: <20211115150931.GA2386342@nvidia.com>
References: <20211109153335.GH1740502@nvidia.com>
        <20211109082042.31cf29c3@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
        <20211109182427.GJ1740502@nvidia.com>
        <YY0G90fJpu/OtF8L@nanopsycho>
        <YY0J8IOLQBBhok2M@unreal>
        <YY4aEFkVuqR+vauw@nanopsycho>
        <YZCqVig9GQi/o1iz@unreal>
        <YZJCdSy+wzqlwrE2@nanopsycho>
        <20211115125359.GM2105516@nvidia.com>
        <YZJx8raQt+FkKaeY@nanopsycho>
        <20211115150931.GA2386342@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Mon, 15 Nov 2021 11:09:31 -0400 Jason Gunthorpe wrote:
> On Mon, Nov 15, 2021 at 03:42:58PM +0100, Jiri Pirko wrote:
> > >Sorry, I don't agree that registering a net notifier in an aux device
> > >probe function is non-standard or wrong.  
> > 
> > Listening to events which happen in different namespaces and react to
> > them is the non-standard behaviour which I refered to. If you would not
> > need to do it, you could just use netns notofier which would solve your
> > issue. You know it.  
> 
> Huh?
> 
> It calls the bog standard
> 
>  register_netdevice_notifier() 
> 
> Like hundreds of other drivers do from their probe functions
> 
> Which does:
> 
> int register_netdevice_notifier(struct notifier_block *nb)
> {
> 	struct net *net;
> 	int err;
> 
> 	/* Close race with setup_net() and cleanup_net() */
> 	down_write(&pernet_ops_rwsem);
> 
> And deadlocks because devlink hols the pernet_ops_rwsem when it
> triggers reload in some paths.
> 
> There is nothing wrong with a driver doing this standard pattern.
> 
> There is only one place in the entire kernel calling the per-ns
> register_netdevice_notifier_dev_net() and it is burred inside another
> part of mlx5 for some reason..
> 
> I believe Parav already looked at using that in rdma and it didn't
> work for some reason I've forgotten. 
> 
> It is not that we care about events in different namespaces, it is
> that rdma, like everything else, doesn't care about namespaces and
> wants events from the netdev no matter where it is located.

devlink now allows drivers to be net ns-aware, and they should 
obey if they declare support.  Can we add a flag / capability 
to devlink and make it an explicit opt-in for drivers who care?