Received-SPF: pass (google.com: domain of linux-kernel+bounces-199386-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <e10398dc-53b7-446f-b22f-f992ba1cc37e@intel.com>
Date: Mon, 3 Jun 2024 08:47:56 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] PCI: vmd: Create domain symlink before
 pci_bus_add_devices
To: Jiwei Sun <sjiwei@163.com>, <nirmal.patel@linux.intel.com>,
	<jonathan.derrick@linux.dev>
CC: <lpieralisi@kernel.org>, <kw@linux.com>, <robh@kernel.org>,
	<bhelgaas@google.com>, <linux-pci@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <sunjw10@lenovo.com>, <ahuang12@lenovo.com>
References: <20240603140329.7222-1-sjiwei@163.com>
Content-Language: en-US
From: Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
In-Reply-To: <20240603140329.7222-1-sjiwei@163.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RUQza2h0SHdOVkxvTnF6TncwaXkvSTdmaUVrTW1nSzZyNjdkWFhlM1ArSlNV?=
 =?utf-8?B?RCt2UG9HRTJOVWdBcEFvWVdsa1VjMUF4UWJqb1E1QjR6K1BKYkxLWnBpQkxR?=
 =?utf-8?B?SWYrRFRTT1VVZFFVOCttWmdhZzdMT3BDUytxdkF1TlFJT1Y5alhrTkZqZXU2?=
 =?utf-8?B?cm1TV0VKV0Q4dEVwa2tQTFJ0SGt3NkZ5cit6QTUyL1hnZC8vRkFoU2hJaHRJ?=
 =?utf-8?B?ZEFYcm5TMnd3SlZDaTdxZjI4YTI4U3pmU2xWbVdmTStiQmpCa1lUMjNGMUV1?=
 =?utf-8?B?RWYwVmFkSng1N1ZOL013NEljVHZ5bFNSMmpZdFhveUxGb0UwNlRUbVZNS1Bx?=
 =?utf-8?B?bzlFdmdCbzBtNGNDdlA4ckpubkFZRHdYYmtzZnYwV0lQVWhHL0Y4Z1lzSENh?=
 =?utf-8?B?MnhiZm8xckM0VXhxVVFORlFGM280Ym04Q2RIMW1ta2hUcjUrUU5Ob0Qxazkw?=
 =?utf-8?B?UENhcTRoTkEvanNGTS83VE16TWY2cENJMzh5M21CQm5OOVlXWDdHUkhHU01K?=
 =?utf-8?B?ZFIveE9pc2hBRzBMdUJOY2tCSCttL1RacVd3UWJDWWR6UUlISjZtWnZaTWgz?=
 =?utf-8?B?MmhvN2tTOHc2SlRvTExzdzZUcGhEN090WVpqQ2h0UitraE1LTm9hWjdlTStj?=
 =?utf-8?B?T2xXK3ExdlViRFg3SDlocVhmY1NrTklBb3d4cWFuVmpqb1dLNmhMUjI1dEdh?=
 =?utf-8?B?RmdpVExJblpMQ2NXYnJ3MFI1LzdubUVEYlpHblJhMS9xcnl0WkRyV00vOENp?=
 =?utf-8?B?MVQ4VCtGSmhsdDBNT2Y4cXZ3L2ZKWnpMNU1uNVFHSTMrMHZrUnlPNFAxd1V6?=
 =?utf-8?B?OXRad0NEWmljTTdjb0xoZEFFdlZmUVl0dytQelgweWo5WjhrZlJBczdQNGxP?=
 =?utf-8?B?aFBJbVY2cjhUcW1mdmtTaHJUVm9RNU44aUk1SCtXWWJpOGN1dGZhMzFhR2tn?=
 =?utf-8?B?YjBrMjlvTVhIVkVUYWVEQklrd1gzcXhxSERpKzVTUERNd25HTitvbVhLZEU5?=
 =?utf-8?B?NFBKUmhoN3RjMHAvMHBFc2UzaDVaRW16TUVpSmF1WUxHeHQwR21odzhiQ25v?=
 =?utf-8?B?V3JIaGoxMDBWY0JlL3ZpUHdzd3dueVlBMHJnejZMeDU0ajNQUkFZcVhvK0wy?=
 =?utf-8?B?aUFtMTNsZlFGWUVCaFFZTC9EWlVLZUxFOTJqdG5BUVh4V094b09sUmhWOUxK?=
 =?utf-8?B?TjU1TE1rU3BNV1htY3RiMTRUWXl4bDVJYWc1bnVCRmQ0elBmaHU4QWM4MjhY?=
 =?utf-8?B?M1hrbUNwMTc3ajZHSjFHZVgzaGxScjFRNE1vYWZ2b2VWWU05VVRHdy9kdnFI?=
 =?utf-8?B?OG12TnlFQ3NrY1ROZUNrNVNTRnBpQnFWUUhVcnBxZk1ObmNHR2pGS1NueEpW?=
 =?utf-8?B?VEJWaFgzZkRJemZlNTdlL3BhdDBGbjhNOTVQdmJjd25BVmpxZlF6ZjU1Q3Yw?=
 =?utf-8?B?NlRwQlIrS2NaMWkvUVVPcjltdHdlbjFTblJGWTkwTWljalFlNGlMTHZLbGFU?=
 =?utf-8?B?cWRndGZPM1NVbTdxUDVPRDZpOGtZOFBmNUEyTjZPS3BOcy9xZGU4WGNlNEY3?=
 =?utf-8?B?SzUyaVlKYzBRSnJGc0tFakVtZmF6b1RiRzZPQTR0OGFrODhVSUgvWU02Y3Vj?=
 =?utf-8?B?aVh2cXNnSmFHR3kxNFBrU0VYYWdmaWk4SllnUkJBRDRsSnZ4MGV4cUg3Tlpz?=
 =?utf-8?B?YXRrTHI1R2xLL0JXS0JUcDBaaUE3ak5ZK1hQRC9rOXBhQlJaak1ZNW5NVk02?=
 =?utf-8?B?andGSkV2NVpBNHVBNEdieWFKZTRtMW5ZZThOVEFkTmZtc0Z2WjlPc1RJU2p0?=
 =?utf-8?B?Z0FNcGlwVXBHc0liRXhyL1M1ZWw2OEhLbUZ2T2VpRHQwUGVwK3JISkpHQzFi?=
 =?utf-8?B?bmdVaGxqVWJzR2o1VlI4NUN1YjRaMHc0UTExdWlNTlR6TVVrdkZCM2dZRTNz?=
 =?utf-8?B?YjlIbkd3ZHUwdlA2dVVNR3l1YVRSalZQeFZER3Y5MXFtaktOZ2xBTDdTU3RP?=
 =?utf-8?B?TGUyQ0QxYkhKb3kzencyM1FlS1U0cmxZYjBqbjlYWXA0VDdzWmdheWtkUHUz?=
 =?utf-8?B?Q2xkMVl2b21UUlgrUmMxYzN6SzNSZThMdStWNElvWmUyMStYVklNa2FUZyta?=
 =?utf-8?B?SnI5WkJzZ2M1Mnc1NitLamNHN3J3RmxsYTJDclFreklLejlFbys5b0Erdkpo?=
 =?utf-8?B?VEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f5c528d4-7ba7-4c55-0817-08dc83e48b4e
X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5636.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2024 15:48:03.7172
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7GEBifjaFRTfprQmjQ5hRk/AcotrAI3uapKXzl4onCyuoddEpPFtStVm1XgjauLB6lFnzcdwPyXbbWGDf17Te9uhoy7BIVoaEfSiKLscntM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7337
X-OriginatorOrg: intel.com

On 6/3/2024 7:03 AM, Jiwei Sun wrote:
> From: Jiwei Sun <sunjw10@lenovo.com>
> 
> During booting into the kernel, the following error message appears:
> 
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device''
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.'
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1'
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.'
>    (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.
> 
> This symptom prevents the OS from booting successfully.
> 

I'm just curious: has this been doing this forever or has this just 
started recently?

Paul

> After a NVMe disk is probed/added by the nvme driver, the udevd executes
> some rule scripts by invoking mdadm command to detect if there is a
> mdraid associated with this NVMe disk. The mdadm determines if one
> NVMe devce is connected to a particular VMD domain by checking the
> domain symlink. Here is the root cause:
> 
> Thread A                   Thread B             Thread mdadm
> vmd_enable_domain
>    pci_bus_add_devices
>      __driver_probe_device
>       ...
>       work_on_cpu
>         schedule_work_on
>         : wakeup Thread B
>                             nvme_probe
>                             : wakeup scan_work
>                               to scan nvme disk
>                               and add nvme disk
>                               then wakeup udevd
>                                                  : udevd executes
>                                                    mdadm command
>         flush_work                               main
>         : wait for nvme_probe done                ...
>      __driver_probe_device                        find_driver_devices
>      : probe next nvme device                     : 1) Detect the domain
>      ...                                            symlink; 2) Find the
>      ...                                            domain symlink from
>      ...                                            vmd sysfs; 3) The
>      ...                                            domain symlink is not
>      ...                                            created yet, failed
>    sysfs_create_link
>    : create domain symlink
> 
> sysfs_create_link is invoked at the end of vmd_enable_domain. However,
> this implementation introduces a timing issue, where mdadm might fail
> to retrieve the vmd symlink path because the symlink has not been
> created yet.
> 
> Fix the issue by creating VMD domain symlinks before invoking
> pci_bus_add_devices.
> 
> Signed-off-by: Jiwei Sun <sunjw10@lenovo.com>
> Suggested-by: Adrian Huang <ahuang12@lenovo.com>
> ---
>   drivers/pci/controller/vmd.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index 87b7856f375a..3f208c5f9ec9 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -961,12 +961,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>   	list_for_each_entry(child, &vmd->bus->children, node)
>   		pcie_bus_configure_settings(child);
>   
> +	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
> +			       "domain"), "Can't create symlink to domain\n");
> +
>   	pci_bus_add_devices(vmd->bus);
>   
>   	vmd_acpi_end();
> -
> -	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
> -			       "domain"), "Can't create symlink to domain\n");
>   	return 0;
>   }
>