Received-SPF: pass (google.com: domain of linux-kernel+bounces-201219-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <c1885394-9edf-47a7-a4f8-1e456ba52316@intel.com>
Date: Tue, 4 Jun 2024 11:00:59 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2] PCI: vmd: Create domain symlink before
 pci_bus_add_devices()
To: Jiwei Sun <sjiwei@163.com>, <nirmal.patel@linux.intel.com>,
	<jonathan.derrick@linux.dev>
CC: <lpieralisi@kernel.org>, <kw@linux.com>, <robh@kernel.org>,
	<bhelgaas@google.com>, <linux-pci@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <sunjw10@lenovo.com>, <ahuang12@lenovo.com>
References: <20240604135153.9182-1-sjiwei@163.com>
Content-Language: en-US
From: Paul M Stillwell Jr <paul.m.stillwell.jr@intel.com>
In-Reply-To: <20240604135153.9182-1-sjiwei@163.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aUIwUE92Y1lnK3N4TDZLbkExYWJEc3BwcVd1aXQwaFRPL2liSS8xNUpDaC9r?=
 =?utf-8?B?SGFOQlQzR0d5bnNTSlNUT2R4bkkzN1ZPeTBiV3d5SHJEcjBsVkNGWVFJbkdY?=
 =?utf-8?B?N3VWVERpRTVjaGtSSzlGMnBIU2ZORmg0cnBocDl4SzFaY0YyS1B2Z1R5OCtF?=
 =?utf-8?B?elA3SjhFaFRwbkFmTnd4YkVjV0dIbjFmbjNKSG1FM1dpNk1yK095ZnNQeTRO?=
 =?utf-8?B?TnNZQk9qUUN2NTYzdkpqa2p0QWhzWHpETlNCa3pKdmQvMnlucit1ZTl0UG1k?=
 =?utf-8?B?dkgyaFM4dFZERjdGWGFaTzlmSlhqUnErL3JFOHVMSGpiN2lDMlJlMUlKL0l0?=
 =?utf-8?B?SFY3VVRZU1FGQUxCaEtUN05EMFY2UkRyMVlJYm5TeDlUWU5URnhISGtXNy80?=
 =?utf-8?B?cXhEcTRIR0wxMW5OYjAyUXZqWUtzZ3lmM01CZHNFUm96UjlTSHlOa1lNaTlz?=
 =?utf-8?B?dGU2bElHSjJYZlBXTDdPSVFmK1RXbmZ6K01VMDJybHJvOGRoWEFQWHQ5NzJF?=
 =?utf-8?B?ZTMxbHBvZlc5MUdVbXNQUGNXamtKNkg4YmVjNDZjZnJiaFVVdTVHUi9wRVJX?=
 =?utf-8?B?dDJqZXN3RWtpeWFVYk92LzVFbG4ycEYvbnFXTzdkZmo3b0JHRHlnOGx1OE1Q?=
 =?utf-8?B?MnJ2ZmJ1bElmMXh1L2xGR01NOEprMmswZ2ZPOXlCLy9ucWYvUU1KM1FQYzZo?=
 =?utf-8?B?VDJMVGd3REdIUEh5L24wdmFvSHpaZnVSNFpoY0FGMFZHMHhhUzJuVDk3alll?=
 =?utf-8?B?YkIvdmNwbExnOURFNGFCSUE1amZvckhwNWFaQVNuMXUxdFFING54Sng4ZjBl?=
 =?utf-8?B?bDdwdDkxcnVoTS8zT3FOMStLMXF3Q0RBOVJIUXFRUmFtR3VzQ0x3ZmFRU0ZH?=
 =?utf-8?B?SnAvYXhMMDJyYnhwSENxMm81V1kzSDBiS0V6dnZCWlhscDFFajREaUoyeXRH?=
 =?utf-8?B?SW1ZbzZsNGRzVXJRT2w1NDBZTHk5UlhNOVF2WDdScGUxc1FDL3ZpMnZsclBB?=
 =?utf-8?B?TktTWnZKM2pBUkV6SVczc2FWaEFDT0FNMjNBdTdhcFRsTmlJcmxIODI2Sk5k?=
 =?utf-8?B?NVJONWI1OVFnVCtTK09MUDhwVHRVS2ZiMGtGS05Ba3dBbWwvMFBRSFBmOCtE?=
 =?utf-8?B?NUY5Um5vbk9yNlV6R2FJVXNDcXBLM3k0ay9IZzhoYTdRN1lGM1NZRVBxRTVJ?=
 =?utf-8?B?NG14RTJNbXNKNi9SNDBrL2s0QnJCbFRqVTYzMVJ6U3JMTkRzMS91ZmcySXND?=
 =?utf-8?B?NGp6TTFXeTNsK01PdEdiYnVHZlQrd0preEFpcVJJb1pVN05QakFweUM5WVRF?=
 =?utf-8?B?UDdVSldvUGJKemNHdVJvY0RMZ0w1Y3VYRHdxRG9ZV0doS3I1UUYyYUxoWEg4?=
 =?utf-8?B?NzY5a3pNUHZnZTFzdWY4K0ZTRmNHWlYydXJqc1l1M0o3b1RXdC9hbGZDeFhB?=
 =?utf-8?B?aFh3WGpVNGxJY2JnNUd3OTAydzdvODlvTEorT0NCb3ZtWC92QWcxdkFPOXI2?=
 =?utf-8?B?eWNKWkhqazRLcS9ucXVHOWxsckhKeTViaXo4Y1lxZWxqTzhqbXVtTjZwZG5W?=
 =?utf-8?B?aFp1NDI3WVVnR2Vaa0tDb3NhdkxsVENlWVFNMUlxT2c5L1ZoTXJySzJnZjEv?=
 =?utf-8?B?WnBpK3NKSlM4enZGeGVraEQwY0ZiaFA2S2Y2KzEzUVdpMktSTmtaeG0rYWUr?=
 =?utf-8?B?TTFZeFJ5eCt2WFpoWXhOZ1VvTmZGM29MQ0o1TmQ4OXA1NVdGL01wTWk5bVZZ?=
 =?utf-8?B?TFQ4bzVXT1NkbGdxclRHdy9nQkp3eDlzb2hQUktpTUYzcmlOc0JPWlk1WDRi?=
 =?utf-8?B?Y0IySWVsOWtJVHZOR1VaamxpQ0ZlVWN2Mk4zdFF5RGJtTGcxaVJoR01hMFFu?=
 =?utf-8?B?L0ZYcjROZnEwLzgyMkhCM3p4ejI5VVZScUszdmd6ZDN4VVZyNDNmMDZDWk93?=
 =?utf-8?B?WVE3UTZIeWdpbGZyUTZ3bk9EWjl0em5oYVhkaTBrY0RVb2l1dEpPWTkwVWZR?=
 =?utf-8?B?L1lCTTRzSmxlanRRcStaR2hvWU1TQUlKZXhBbmxwaWs1NndnYXBWTmU4NVFs?=
 =?utf-8?B?clhGWFp0ajlnYWJVLzNUc0s2OTlvdFJaSll0Z2I3UlJ1UTJPTi9OTnYvMzJJ?=
 =?utf-8?B?Zi9RTW00ckgwa1NTdjhKM1ZFYysyRFpoS2pkZzBLYUpnNUpzUGlOMVZ1Y053?=
 =?utf-8?B?MkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e03c148a-e009-4808-33e3-08dc84c04c26
X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5636.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 18:01:02.8114
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5R2htJvoz7BWOvBKi2mUC+w1rG+7eijoxR3eVkp5/7sJnww6qzZAaDAfk2Jf9bRmV4tkL6f+bLQepdi1/jtheO5EOVjMbSJWEkaJa9shXEk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7853
X-OriginatorOrg: intel.com

On 6/4/2024 6:51 AM, Jiwei Sun wrote:
> From: Jiwei Sun <sunjw10@lenovo.com>
> 
> During booting into the kernel, the following error message appears:
> 
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device''
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.'
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1'
>    (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.'
>    (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.
> 
> This symptom prevents the OS from booting successfully.
> 
> After a NVMe disk is probed/added by the nvme driver, the udevd executes
> some rule scripts by invoking mdadm command to detect if there is a
> mdraid associated with this NVMe disk. The mdadm determines if one
> NVMe devce is connected to a particular VMD domain by checking the
> domain symlink. Here is the root cause:
> 
> Thread A                   Thread B             Thread mdadm
> vmd_enable_domain
>    pci_bus_add_devices
>      __driver_probe_device
>       ...
>       work_on_cpu
>         schedule_work_on
>         : wakeup Thread B
>                             nvme_probe
>                             : wakeup scan_work
>                               to scan nvme disk
>                               and add nvme disk
>                               then wakeup udevd
>                                                  : udevd executes
>                                                    mdadm command
>         flush_work                               main
>         : wait for nvme_probe done                ...
>      __driver_probe_device                        find_driver_devices
>      : probe next nvme device                     : 1) Detect the domain
>      ...                                            symlink; 2) Find the
>      ...                                            domain symlink from
>      ...                                            vmd sysfs; 3) The
>      ...                                            domain symlink is not
>      ...                                            created yet, failed
>    sysfs_create_link
>    : create domain symlink
> 
> sysfs_create_link() is invoked at the end of vmd_enable_domain().
> However, this implementation introduces a timing issue, where mdadm
> might fail to retrieve the vmd symlink path because the symlink has not
> been created yet.
> 
> Fix the issue by creating VMD domain symlinks before invoking
> pci_bus_add_devices().
> 
> Signed-off-by: Jiwei Sun <sunjw10@lenovo.com>
> Suggested-by: Adrian Huang <ahuang12@lenovo.com>
> ---
> v2 changes:
>   - Add "()" after function names in subject and commit log
>   - Move sysfs_create_link() after vmd_attach_resources()
> 
> ---
>   drivers/pci/controller/vmd.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index 87b7856f375a..d0e33e798bb9 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -925,6 +925,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>   		dev_set_msi_domain(&vmd->bus->dev,
>   				   dev_get_msi_domain(&vmd->dev->dev));
>   
> +	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
> +			       "domain"), "Can't create symlink to domain\n");
> +

I think you should move the sysfs_remove_link() line in vmd_remove() 
down as well.

Paul

>   	vmd_acpi_begin();
>   
>   	pci_scan_child_bus(vmd->bus);
> @@ -964,9 +967,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>   	pci_bus_add_devices(vmd->bus);
>   
>   	vmd_acpi_end();
> -
> -	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
> -			       "domain"), "Can't create symlink to domain\n");
>   	return 0;
>   }
>