Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sun, 18 Dec 2022 11:53:23 +0200
From:   Leon Romanovsky <leon@kernel.org>
To:     Petr Pavlu <petr.pavlu@suse.com>
Cc:     tariqt@nvidia.com, yishaih@nvidia.com, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: Part of devices not initialized with mlx4
Message-ID: <Y57jE03Rmr7wphlj@unreal>
References: <0a361ac2-c6bd-2b18-4841-b1b991f0635e@suse.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0a361ac2-c6bd-2b18-4841-b1b991f0635e@suse.com>
Precedence: bulk

On Thu, Dec 15, 2022 at 10:51:15AM +0100, Petr Pavlu wrote:
> Hello,
> 
> We have seen an issue when some of ConnectX-3 devices are not initialized
> when mlx4 drivers are a part of initrd.

<...>

> * Systemd stops running services and then sends SIGTERM to "unmanaged" tasks
>   on the system to terminate them too. This includes the modprobe task.
> * Initialization of mlx4_en is interrupted in the middle of its init function.

And why do you think that this systemd behaviour is correct one?

>   The module remains inserted but only some eth devices are initialized and
>   operational.

<...>

> One idea how to address this issue is to model the mlx4 drivers using an
> auxiliary bus, similar to how the same conversion was already done in mlx5.
> This leaves all module loads to udevd which better integrates with the systemd
> processing and a load of mlx4_en doesn't get interrupted.
> 
> My incomplete patches implementing this idea are available at:
> https://github.com/petrpavlu/linux/commits/bsc1187236-wip-v1
> 
> The rework turned out to be not exactly straightforward and would need more
> effort.

Right, I didn't see any ROI of converting mlx4 to aux bus.

> 
> I realize mlx4 is only used for ConnectX-3 and older hardware. I wonder then
> if this kind of rework would be suitable and something to proceed with, or if
> some simpler idea how to address the described issue would be better and
> preferread.

Will it help if you move mlx4_en to rootfs?

Thanks

> 
> Thank you,
> Petr
>