Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e939dbde-8905-fc98-5717-c555e05b708d@suse.com>
Date:   Mon, 2 Jan 2023 11:33:15 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Subject: Re: Part of devices not initialized with mlx4
To:     Leon Romanovsky <leon@kernel.org>
Cc:     tariqt@nvidia.com, yishaih@nvidia.com, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <0a361ac2-c6bd-2b18-4841-b1b991f0635e@suse.com>
 <Y57jE03Rmr7wphlj@unreal>
Content-Language: en-US
From:   Petr Pavlu <petr.pavlu@suse.com>
In-Reply-To: <Y57jE03Rmr7wphlj@unreal>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SFBnSytGWWtlQmxWZVU2N0wyaXlueU9rQ2V5Z21HSXc3cVl6RFcyZGo1THpJ?=
 =?utf-8?B?Vkp2aXREM2JZNkdOQzY5VHpSSm1VZjFvbk1jYjZ1ajdWWG9vRkdtdlNOeTdN?=
 =?utf-8?B?NXQ2YURBK3dpMENGeGhWa1poenpDWG1YTEFyQWlmeWZqZ29oc2RzRjJrbDR1?=
 =?utf-8?B?eE4yOVM1Y25xS3A1SGxoYnF6UTBQRVV1K0tVT1JKZkRJVU1HVlRadXNDY3hv?=
 =?utf-8?B?czNuRFpibFNiMEVROThKcWk0OEJxUWNaeFhLTzRzKzdFVHEyQ1I4RjFvMGQ0?=
 =?utf-8?B?T2szeHhhaDRlM3pCcVJwNHBlNzIrdU9EYk51UkRSd0NDSENSN2VQV3ltWTA4?=
 =?utf-8?B?UDM5Zm9HbGQrQUNYa2w0K29YaGRHb1BaeUc1dUFiZ3d5d0NCZHVCREJacXhV?=
 =?utf-8?B?dE80Y3o2S2ZWR21xdTZTMTc0bjNZZEVhUFExNXR1TGlhSzNzQjhUWTVrZUY5?=
 =?utf-8?B?amcvN2VkNndnSkwyZS9FRFVYWlZhWGl0ZFJxU3F6L3AwWDNVZ2U3blpOb1Zy?=
 =?utf-8?B?cVpkMHlDUHZDaXdiQWpZOXFYUDRaQzlUQXBPeWc1bTBXYWdDUzYyVkRhQXkv?=
 =?utf-8?B?S1pJb1hQVDYwVkZ4S2dtOXN0MzcxbUh5WlM2TGhnbFhpamNmRFBRcklmN3pV?=
 =?utf-8?B?SE9YZ3liUm5oRDN5TEVtVUtkelRrZVpjVjJkYmx4QkVWUmtKeGZvT1Y1Qmph?=
 =?utf-8?B?cnY1K2wyOFlvdVNGQVQwN0plZ1creHArWURlT3ozNFF3ZnZrVi81Q0dZMDdv?=
 =?utf-8?B?SmgxMWUzczdQeDZ0RGpNSmVqdkx4ZTJuS0NsNkpvRUtRdEhLdnB6Q29QdzBN?=
 =?utf-8?B?RUVkc1IrRW45MU0rWDBZdDY0a0xLVW1BL0VJUDRERUR0S2g1V2Jub1FEZ1Yx?=
 =?utf-8?B?SDBtblJzZXV6RElKRmMyRmhkVG5QS0xjc1Y0azQ0L0M2MUtydmQyODE2bUlu?=
 =?utf-8?B?OVZuYUZGd1R5bE50QWFXckxXU2p4NVhTcmNWWmI5RTZ4clNxWXJ1eTB5ejV0?=
 =?utf-8?B?MHZSRDQrTUJ4b3dJbUthMzUyZ1d3V2Naa044UnNXTzcrRlM0VWlVdzVVMXMz?=
 =?utf-8?B?VHpwU2QyMnQzVEdCbEhENWRhOTR2d0tpR0tHdDhVSFJCSUZUbVVyaEc4RUlR?=
 =?utf-8?B?NkN6SkZ3dkZldGM4QjFpY3BqL1ozbUh6U2EzL09tanlaYm9aNFg1MnNVTUw3?=
 =?utf-8?B?Rm1JTzFPYVE3dHdRUkN5SWRidjRUK3lyVkUvYXhROXJYNFN3bDNXR1ZFazhj?=
 =?utf-8?B?V2pQRDdqelBNRXc5MlFXYk5xMi9KcmFNdDB5WC9ndEJUMmVQb0I2VGczNmlV?=
 =?utf-8?B?SDBUTGZPaVBZbEFJc3Z4ZldJRzdvNW1wVHRvTWZIZ0tKQjVVUWR2RHRjNm13?=
 =?utf-8?B?QVNKREVYZ3A2SGtxVHNPZkpMTkNJeUdKN29YWERZbkhCWHcxUlNSNkFhblpH?=
 =?utf-8?B?bXFFTXdSTEk3UUVMcVNZZFR1blpld1VMcTg4OXc1VzRwVGZRVVBIOWIzZU1r?=
 =?utf-8?B?MDBjdjRaZW5QQkk0VnBmMzVVdUhnVFVSaGlpcFJERkJYMDRJSmxlajJJZU9T?=
 =?utf-8?B?dW93bWR6NEhnNS9UWXVPZmFEMlpYMjEvVWhDYndiVWZPRTI1UVF1c3Y3U3ZZ?=
 =?utf-8?B?ZUJHU09VUUdUNzZ6a3NYMUNWWXhTUkxkTk1DNCtPVnVuVkJqVFV3YmhSZXhI?=
 =?utf-8?B?U0NXdVAySFA0OXpwbmVFTzgzeFV4TkpQUGtRQURnYVE2UjdIR3BTL3BKN2d6?=
 =?utf-8?B?TnBUUFlYY2o4SDNaSjVNQThXRXJDRkFtMmZMR2VoY0tFUWJlK0czWDgxMnJy?=
 =?utf-8?B?RVoxdUh2eVEwRWdVR211V1U2aDRreGZ6bC9BVTZ6L2VxUFluWXB5YlJINndj?=
 =?utf-8?B?blpzOTE4UXFwQWpKV2JrWEdkMmFSR3hEaEZkV3dLVzh6NmxRSHhCV294dUlN?=
 =?utf-8?B?QnRkenNkVjdNd1FCUVlnUWN0dC9WVkR6R3JlQlhaWklhbWU0dXNnQ29UbHpK?=
 =?utf-8?B?RFhYWTlqVVIwQ0RKWjRNNUJONUlYZ252QzNXMGxRdDFZZW03L1pwUFlvTFlV?=
 =?utf-8?B?SG0wTmFWUVBtbU0xSTJyalpFcTNwdzZENkdJZ1BXZERiaGV2ZkVCZmRycGpQ?=
 =?utf-8?Q?ABXwSyfgbAAzYaJCyu3uOxDFG?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a50a057-6c73-44b6-2048-08daecaccf7c
X-MS-Exchange-CrossTenant-AuthSource: AM0PR0402MB3395.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jan 2023 10:34:19.5553
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: J4lfbIbx1vB+gsFEdJxEIkmluAvaZLjRhnzhDE6hGGIwZXwtv3gOKREJ3OYdIV2R2sEXur0iYUbxEMmkLp0bZQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB7525
X-Spam-Status: No, score=-5.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/18/22 10:53, Leon Romanovsky wrote:
> On Thu, Dec 15, 2022 at 10:51:15AM +0100, Petr Pavlu wrote:
>> Hello,
>>
>> We have seen an issue when some of ConnectX-3 devices are not initialized
>> when mlx4 drivers are a part of initrd.
> 
> <...>
> 
>> * Systemd stops running services and then sends SIGTERM to "unmanaged" tasks
>>   on the system to terminate them too. This includes the modprobe task.
>> * Initialization of mlx4_en is interrupted in the middle of its init function.
> 
> And why do you think that this systemd behaviour is correct one?

My view is that this is an issue between the kernel and initrd/systemd.
Switching the root is a delicate operation and both parts need to carefully
cooperate for it to work correctly.

I think it is generally sensible that systemd tries to terminate any remaining
processes started from the initrd. They would have troubles when the root is
switched under their hands anyway, unless they are specifically prepared for
it. Systemd only skips terminating kthreads and allows to exclude root storage
daemons. A modprobe helper could be excluded from being terminated too but the
problem with the root switch remains.

It looks to me that a good approach is to complete all running module loads
before switching the root and continue with any further loads after the
operation is done. Leaving module loads to udevd assures this, hence the idea
to use an auxiliary bus.

>>   The module remains inserted but only some eth devices are initialized and
>>   operational.
> 
> <...>
> 
>> One idea how to address this issue is to model the mlx4 drivers using an
>> auxiliary bus, similar to how the same conversion was already done in mlx5.
>> This leaves all module loads to udevd which better integrates with the systemd
>> processing and a load of mlx4_en doesn't get interrupted.
>>
>> My incomplete patches implementing this idea are available at:
>> https://github.com/petrpavlu/linux/commits/bsc1187236-wip-v1
>>
>> The rework turned out to be not exactly straightforward and would need more
>> effort.
> 
> Right, I didn't see any ROI of converting mlx4 to aux bus.

I see, but in case you and other maintainers are not immediately opposed to
this conversion idea, I could try to resolve remaining problems in my port and
see how it turns out?

>> I realize mlx4 is only used for ConnectX-3 and older hardware. I wonder then
>> if this kind of rework would be suitable and something to proceed with, or if
>> some simpler idea how to address the described issue would be better and
>> preferread.
> 
> Will it help if you move mlx4_en to rootfs?

Yes, if mlx4 drivers are not in the initrd but only on the rootfs then this
issue is not present. A problem is that VM image templates have their initrd
typically generated as no-hostonly and so include all drivers. Some images
might also require that networking is already available in the initrd for
instance initialization and so must include these drivers.

Thanks,
Petr