Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <0a361ac2-c6bd-2b18-4841-b1b991f0635e@suse.com>
Date:   Thu, 15 Dec 2022 10:51:15 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Content-Language: en-US
To:     tariqt@nvidia.com, yishaih@nvidia.com, netdev@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org
From:   Petr Pavlu <petr.pavlu@suse.com>
Subject: Part of devices not initialized with mlx4
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MFladkcxdXJZekZ0TkdSdkVjN1RQekhSc2hsUFkvZWMybnJrbWZjTy81alpV?=
 =?utf-8?B?THFaeStLc3JMZGF1OXcweHlNWGp0dDd2bkdQUHMrUmJjRlZpYUdtdncwUEVl?=
 =?utf-8?B?SEwxWTU4Wk92cTY5c3FGLzNpaFBNZWdvNGhldVMxdWtXVU55S1Zrc3hWU0Mw?=
 =?utf-8?B?UVhUL0NyQTh3TWNTM2RFWWd2dElDdFFWNzVVOWhmRlQ3eG5VYmhPQjlEQWIv?=
 =?utf-8?B?ekpJVzhTdXU1VWRFRXdHN2FLQUVhK1BDU2lQUjBwTHpLODRWT2dncGNuS3FS?=
 =?utf-8?B?dU1wZnFYUXV6V3BucVNucmN5R3FBcWwwdEdQZHRnTkN1WmRxbCt0ZHRtTFVZ?=
 =?utf-8?B?Z0tuMCt0NDFkSE9LemxQSkZTc1JYMHQ1MXowSEJ3QzhQZTVPRHFrMHJ3NHdM?=
 =?utf-8?B?cDFOMm5RNzZtRGFoaWtiR3JZZUdmOE1RSkVaTVRGRm9qSkJIUGtNMm9jUklO?=
 =?utf-8?B?MFFoYlRCMDBNME1jdVB4RnBLT3c3RjRWaEJkMXBSYkVXckpMQ2xSbEdTb0t4?=
 =?utf-8?B?RlgxL2tKOVN4dTZwWXQ0QXFpUVI0UlRRb05NY1o5UThBVGl4U0IvYkdnSGtB?=
 =?utf-8?B?OHc2N216V0FSQjNXR3V2Szl2eHBLd0ROSmdpdFM3TXdxS211NkVDU3JZa1ZV?=
 =?utf-8?B?dTlZeTdtYWUvNVNtMWpOVjBUZ1R1QzV1Y3FOZjNYTHRWZ0NFdEtJbmc2bFJ4?=
 =?utf-8?B?L3dFUmZoSTJ4RmV5NURHOGo3OXVadnl3QjRDdWdEaWxpZmxXV0NzUldNbzBV?=
 =?utf-8?B?aXQ5ckNUaGVRdy8yaENoTFBramhZZlRYNnROT1lVQktDSzREMTI1ZEJVTzhi?=
 =?utf-8?B?TmRheEU2M2RnUUNONmFFbHhkSFFhZkpBTldPRU42NWloeEY4QXFJcU1GZnh3?=
 =?utf-8?B?NGpxNktmRVYyeUdsV0VFc0NSWFFqcWJPanhIK3QxU0Y5cTZ0S1Y0L2NlMUI5?=
 =?utf-8?B?TkR1SWttdXRXeDM4SnZtUHAxQVdkQ3VnRUtCUk9mU2xtUk9wMzFhQXMrUnVH?=
 =?utf-8?B?RlNqWU9yUkdxVVJkOU5ja2xaYUFLUHRES2VmQWxQMDJhMS93dk1UOG0xY2JP?=
 =?utf-8?B?OTFEaGZzVUV6c2c0S3I2RExQKzFGWkJhV0VHSFdqUiswdzdXM000ZFR6T3Rr?=
 =?utf-8?B?S0lQOGsvMDA4amJWcVJ4VDhnbXFXbzM0TU5ZNnZmM3hJZ1ljNEQvVnJaTi9v?=
 =?utf-8?B?QTNZU29HVVkzVzJycTBiTXUzemIwWVFFSHJCRC9IbWxIR25oYW9sSVVObTB1?=
 =?utf-8?B?V1BvZTBBc25ZdVluUGJpaVhGZEJhRWVTZ0hpRmFxTk91NkY0bXhFR1lldXdr?=
 =?utf-8?B?dGlPRXMwV3k2U290bVA1VUtNcnQ2Y3pBUXp3VG51bHNhYlVPTXQ4emdTSHQ1?=
 =?utf-8?B?TTZNdXNkL0MvYkZ4TWt1cVJTNERiZG5pRVQ0RHoyMC80cG40ZFZmOURjaWFo?=
 =?utf-8?B?Tm9jUUpHcWo5TVZmMGxTaTZEajJNVzZpOXMzMlgyaTZNWEpEdVpXSDNmNjA0?=
 =?utf-8?B?SmhYcHJoQkRNWWNVaUs2NE1IdTZJaDRSQWFZcGJkQ3NVS3pLTDNmMS9WYlFY?=
 =?utf-8?B?WmZXMjF2QVZZcjdDSmxCTERMNVoyTFJIeXplRk5KOHhpZDMxd0kvQ2xwQjda?=
 =?utf-8?B?MmVUcjEzc21iYVgyT2NhSjErT3J0NHlYMHNBQ21RZWZvRHhsVU90TzZFQ1Fl?=
 =?utf-8?B?blBmM3Jhc1ZKRXJDck1WYmFpRVBuR2kxaE5QYVZQNHJNTG9Zd0kxbTI1U1VX?=
 =?utf-8?B?VXkrN1Nrb2JwdlhtNXMyUGNoMW9LWTA2THFYb1YxNkJCek96OUFDQ0RGZVRG?=
 =?utf-8?B?R0ZrM3U5VkpBRmN5dWl6T1g5c0UvZldqL1VUMlVCa1RUU0FPSkp5RU5aaE5H?=
 =?utf-8?B?bHVPK3JxRXNsdHd5V2dTSFdKN0JxUXdsMHZiSDRreDVwckhJZTNFOE41QUF0?=
 =?utf-8?B?Z3BQWnp3TmlzNjNyUkRqYXd0U1JESHJYT3kxTXE3VUJzZFdHR0R3WTZHdklW?=
 =?utf-8?B?Y0tudWdVSDRJOVpnKzNyeFZSZkVpVU5lSzFtQjJ2YmlnUEtMZGdrb1JodXB2?=
 =?utf-8?B?a01LVjNzaENhWHNXU21lZ3pYOVhwUXZ2S2E3a25TNzJnbDZpaGhHSHhGRkZy?=
 =?utf-8?Q?EJ7RANiobkU0sryb1P8FrVz/j?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0082888d-c22d-4948-b951-08dade81e912
X-MS-Exchange-CrossTenant-AuthSource: AM0PR0402MB3395.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2022 09:51:17.0687
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /zqP9WIjs5ZVCJOcPupZOXvKON7nt3DeHGrI5sexaocwIv7mzehB9VpsRfFPYcgM6tHICHMXzQ8tzHYJnFAg8w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB7896
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello,

We have seen an issue when some of ConnectX-3 devices are not initialized
when mlx4 drivers are a part of initrd.

The basic scenario looks as follows:
* A machine has multiple ConnectX-3 devices, they can be VF ones. The system
  uses an initrd driven by dracut+systemd. The initrd is built as no-hostonly
  (think of a VM image) and includes the mlx4 drivers.
* The machine boots. The initrd invokes udevd to start inserting device
  drivers until the root disk is available.
* The udev daemon inserts the mlx4_core driver, which asynchronously requests
  a load of mlx4_en. This is done by calling request_module_nowait() from
  mlx4_request_modules(). The kernel spawns a modprobe userspace task to
  handle this request.
* The modprobe task finds the mlx4_en module and asks the kernel to load it.
  The module loader runs the init function of the module which starts
  iterating over mlx4_core devices and initializing their eth support.
* The root disk becomes available in the meantime and the initrd logic starts
  the switch root process.
* Systemd stops running services and then sends SIGTERM to "unmanaged" tasks
  on the system to terminate them too. This includes the modprobe task.
* Initialization of mlx4_en is interrupted in the middle of its init function.
  The module remains inserted but only some eth devices are initialized and
  operational.

The modprobe task uses the default SIGTERM handling and so this signal becomes
fatal. Specifically, it causes the create_singlethread_workqueue() call in
mlx4_en_add() to error out. The workqueue requires a rescuer thread and a wait
on the new thread fails because a fatal signal is pending.

As mentioned, this can result in only a part of all devices being initialized.
It could also likely happen that the modprobe task fails in some other obscure
way as it has its root switched under its hands. It is a task that is
completely asynchronous from any systemd control.

Has anyone else seen this issue before too?

Note that some parts of the problem are not fully clear to me yet. In
particular, systemd also sends SIGSTOP before and SIGCONT after the mentioned
SIGTERM signal, which can actually in some cases prevent the kernel from
treating SIGTERM immediately as a fatal signal. I'm waiting on some additional
test machine to analyze this part more.

One idea how to address this issue is to model the mlx4 drivers using an
auxiliary bus, similar to how the same conversion was already done in mlx5.
This leaves all module loads to udevd which better integrates with the systemd
processing and a load of mlx4_en doesn't get interrupted.

My incomplete patches implementing this idea are available at:
https://github.com/petrpavlu/linux/commits/bsc1187236-wip-v1

The rework turned out to be not exactly straightforward and would need more
effort.

I realize mlx4 is only used for ConnectX-3 and older hardware. I wonder then
if this kind of rework would be suitable and something to proceed with, or if
some simpler idea how to address the described issue would be better and
preferred.

Thank you,
Petr