Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Message-ID: <a7e8e2cc-a228-d444-4266-35711d220143@oracle.com>
Date:   Wed, 13 Dec 2023 09:25:03 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [RFC 2/2] PCI: acpiphp: slowdown hotplug if hotplugging multiple
 devices at a time
To:     Igor Mammedov <imammedo@redhat.com>
Cc:     linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org,
        mst@redhat.com, rafael@kernel.org, lenb@kernel.org,
        bhelgaas@google.com, mika.westerberg@linux.intel.com,
        boris.ostrovsky@oracle.com, joe.jin@oracle.com,
        stable@vger.kernel.org, Fiona Ebner <f.ebner@proxmox.com>,
        Thomas Lamprecht <t.lamprecht@proxmox.com>,
        linux-kernel@vger.kernel.org
References: <20231213003614.1648343-1-imammedo@redhat.com>
 <20231213003614.1648343-3-imammedo@redhat.com>
 <d93c4614-1bbc-3a30-305e-28ff75d7fde2@oracle.com>
 <20231213110556.5f1d83bf@imammedo.users.ipa.redhat.com>
Content-Language: en-US
From:   Dongli Zhang <dongli.zhang@oracle.com>
In-Reply-To: <20231213110556.5f1d83bf@imammedo.users.ipa.redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?enREVEZhekhmNVNNQi9udGUrZjNKVTcvT2g0Y2xIOXM2VW5iQ3E0ZFdZbmF5?=
 =?utf-8?B?bmFtZFJvTmx0bVE4UW1RZ0FHSk9QRmlPdE1wck5MaVg0U3Fwald4VWM4MTdF?=
 =?utf-8?B?N1RJMW9aQ3plUlNCOGtpaHhpQ1hWSmRBVnNKR2pvTnFoQitLZ0ZieUYxZ3gx?=
 =?utf-8?B?TVBrNmlPNmtuQVZFY3JrS216UUNYdHdzRkhVN0g1aGJCcEFPVFVQVFhkaTd6?=
 =?utf-8?B?WVpoU21sUUFXUEl6amRvS2tJVzNJcnlBc3RmelFXc3VHbGNiVTduYVJsRVN6?=
 =?utf-8?B?U1VBNUdkTkYzcHA4K3p1em11bnJ0K3dKUGhYMi9EYXRlbEk1THJoTUk0aW5F?=
 =?utf-8?B?ZWp6c210a3k2bjNTaitmamJJdXRSSEcwSTJrdDlPeWMvZVh0QWp0N2pXaFoy?=
 =?utf-8?B?dy90ZE5Ebm5ybXNmdWtnTU5YMWZLcFJhWUZjUTgyUlZpUmF1aGIyL0toSXQw?=
 =?utf-8?B?Z0wzellsUjk5U20rNHRETkVEZXFYQnYzSjRzVjhvUTBFVTE3TEw4ZHhqb2c3?=
 =?utf-8?B?TFRHc0U4NFg5Qzd3WFBOYUVnZHNUQ0lQSm53WkorWVM2MEpiRGxyN2V3YWF4?=
 =?utf-8?B?YUNjUTZxVjFTZGtvdFB6d1FlQkdDSzVFRUx1b2pQQzhJa1l1QnlJN3BMS1VW?=
 =?utf-8?B?bVZ2dFdiaDhIWVNmK2dmM2ZnM2p2K2tyaTBzMGEwVXcyb2RtbUMya0NyUmxy?=
 =?utf-8?B?dWUvWTdNb0VLTmlPWS9KMzJYLzVMMGd6U0U0ZWlGa2lZMjBFVEZscm9XVm5l?=
 =?utf-8?B?N2t4UmxTUGxYZ09UdGZjSFhWQi9Pazd3R0hkQjBObUlnekdoeW5HYmZUWDk3?=
 =?utf-8?B?WG5QcnJCSUlqUU53K0R0UktzN25xRlZraTRXaXZzZzVJUWNFOWM4ZVZML3NK?=
 =?utf-8?B?UVhodUNhd1pmOGQ3czh4YUhGSXZITHU2dGhKazFXTGlmM2JFVHpSM0NpNCtj?=
 =?utf-8?B?VkNJVlRXMGc0SlhvRnZyOUt6akVUTzNlUHNNQjYzVm5pck93amZVQnJ1aCtU?=
 =?utf-8?B?djZtVjNvV0ZnbnI2cWhuekRVbEVvaVBDVk4rT01BbUhRYlZydUM0SDllZVhM?=
 =?utf-8?B?aU9MR0N0T0tyUHFLeG55V0RENkx5elp2bEVoSDdKOVNiQUFZbjAyMkFyT1N2?=
 =?utf-8?B?NHdBN0lCTnYzVC8yS1V6L3RxNlluQ3ZQajcrN0ZrYUY5dGJ6eXBmUko2cEVI?=
 =?utf-8?B?Tk9rbUFPRWNNWXZEYThXbi9VdXdoVzdpWHovWVovUWdDY3ZuNDJJQ0NhQXNE?=
 =?utf-8?B?YjBkSjNIWjdWVXB6RVVXemhsQ1V5MzhyOUNqUzVJTTJvRCtRMW5ZVUVtOFBQ?=
 =?utf-8?B?SUFURjVqd0Y1TW9FM1NlYnByWTUwS205WVZ3OW1abEl4ejJUc3plNTZ2QUM1?=
 =?utf-8?B?L3Rob1hidGtHdDBQRDV2aFN2YWtTMnlUZWZzOGdFdnFoZVRhYXk5VytxODJ1?=
 =?utf-8?B?S1FxcGh4S2xqUG5LUG5wckY5akZnTE03VzViS1MzNnVBbW9WOTBwK3Nab1JS?=
 =?utf-8?B?cExkcG16QUtGNFhEY1R1d2JmY3NYVmQ1YVZ2amk3RlFTbFFjc01oOW1wclRE?=
 =?utf-8?B?YVBycTd1QW12Vlh3d3RnZTcxZGg2Z2pEL3RoOGhxQWIwTkFJeW9MelRiSGd2?=
 =?utf-8?B?aHdkYmhIdXJCS010WlI0cnJrQlhEYTRHTXhtaDgwNkFOeWgzZjgxWjVIQUV6?=
 =?utf-8?B?UVlQY2lmQWM2Mm95M2R4SWlrSUEvRmFwcUVtZWlHeUUzYXBpeGNMMFR2TWVm?=
 =?utf-8?B?T2VVRGlhWHdNenBlajdZbWdkUzZSUkYzbWU2ajZnYnU3YzVRYjFicmxHVmZH?=
 =?utf-8?B?TTNwT0Z3VlhzeWpLS2xLMURuSEtCdUQwNUoxdXlNMjNxV1ZKZDhyeUtJeVVm?=
 =?utf-8?B?Y25QUElsd2xiWHZZMHVVMXhGN245dnlyODJXcGZXZXNRcmR3VWlNVHZGVXdC?=
 =?utf-8?B?YmkwdUZqT01zZmw4cEpMbFMzZ2JCUi9VM2kxVmhvZVgyN29RMEtSN05SdkFo?=
 =?utf-8?B?Zzl1ZE1qU2xDcEFEQlpvNm9SbUJyY2tobkNvZjVxelc2UTl0VjFyU0NkcXNE?=
 =?utf-8?B?OHgweWQ1TjJmUzZJMEFBK0Q3TVJwS1B0bGRobjV6ODBjRk9VeW03ZXFiaGIr?=
 =?utf-8?B?a3JvTFBub0o0NnhsWEJ4T0U3L3Y5MTBaRitnUjEvK3M5WUc0Y0lseHBPcFpT?=
 =?utf-8?B?Mmc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Bt8jhSnP8jenJN5CbAyty/aGxOeeK7iKBej3IwFVUWdGNY7Ttbb7UkUHX6rMvCO0Y0ntm3qNF25W1iIolyAQ0zag84gHcXgVOrkGaMszZ1OFQ/FDpkKtdpE/xIQRMOGHUiEQbn87/tgJ7Ik2vcWbzDOoQ2tsyPj7kHTMOX8fknk1Vh24WeKnkjtouGS6zMhZWwFhsJqNGUq14us22G2U2OtPS6IUtvNf8PfhzN0onHKDpS1h1O2g6LYLVTdyVYO2LVAtTi/v3gfxvX2yZOauCk0ponZltuaxKjVLsFG1JIFvJLClMQetPiWer0Iz6zfeDnCNYS3rsSW+SXUMRiofDfrEwe+3efaVwVQhKRCZWBXSTCUp9oyx/ngcpCpguDOKEx1yDRxkMaktXjXJJGeaFvoKRvRFns2/U7u7ul5+u0rnYSfcvjuTrrrzuUkYnp9+flgrAP6lWuV7Wham5rU09PaVhaRKwvxpdtE51Tzm5wdyb+lydmNMKivwmjDEBWDcI0+Yq3tZPFQKB8cfmmdbNy+gKS9Fp+21jcaoDkGqJfVjS4QThJytWPOaWbI0/DjFCxZhzOeoT8c7ZPFLOrFn9JS7BIYQto5xKOXocYG6lxm9ftWROKW5SXYjUc1YYnQC0SSa/ShU9KQWgZV05gfnf5Ao9AiVRMDSP6Sp9WxOJhJxddZHbSB19K2pizgpvPNS/u2WEIXATJ13jAbeEJJlset/flqtylmi5jUBk2ulsXAtIDSiiYksPQNnjWeUB2+NoGTsf6ujQtgqKjFgSRkCDWG377lC25XWS2tckiEnokElLnqbyecyFU9FXfwJsqdTZftUMMKNmLq+3/NGXZ5NzNcAwFEzRM6INYkiHF33CVHQzHSHCOutceQIefNbCtC/htxGmR3oKCVKWxo7VlnFvIQhRUpEI03JYFp3mY/saEc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 636b8a66-27fc-40d7-5ca0-08dbfc0071d6
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2023 17:25:04.4394
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Tr5iv4v/weKTp7gOYsn4JuMX4+E8HTQTONSgrXBU8giKjbFOoEQhc8U+NCBDqe2grr33J2vnqY5JZ54OIY1NYw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB5818
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-13_10,2023-12-13_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0
 phishscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312130124
X-Proofpoint-GUID: p6TPaBfsate2sBMavMQq17usVZQhM-lq
X-Proofpoint-ORIG-GUID: p6TPaBfsate2sBMavMQq17usVZQhM-lq
X-Spam-Status: No, score=-1.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 13 Dec 2023 09:25:30 -0800 (PST)

Hi Igor,

On 12/13/23 02:05, Igor Mammedov wrote:
> On Wed, 13 Dec 2023 00:13:37 -0800
> Dongli Zhang <dongli.zhang@oracle.com> wrote:
> 
>> Hi Igor,
>>
>>
>> On 12/12/23 16:36, Igor Mammedov wrote:
>>> previous commit ("PCI: acpiphp: enable slot only if it hasn't been enabled already"
>>> introduced a workaround to avoid a race between SCSI_SCAN_ASYNC job and
>>> bridge reconfiguration in case of single HBA hotplug.
>>> However in virt environment it's possible to pause machine hotplug several
>>> HBAs and let machine run. That can hit the same race when 2nd hotplugged  
>>
>> Would you mind helping explain what does "pause machine hotplug several HBAs and
>> let machine run" indicate?
> 
> qemu example would be:
> {qemu) stop
> (qemu) device_add device_add vhost-scsi-pci,wwpn=naa.5001405324af0985,id=vhost01,bus=bridge1,addr=8
> (qemu) device_add vhost-scsi-pci,wwpn=naa.5001405324af0986,id=vhost02,bus=bridge1,addr=0
> (qemu) cont
> 
> this way when machine continues to run acpiphp code will see 2 HBAs at once
> and try to process one right after another. So [1/2] patch is not enough
> to cover above case, and hence the same hack SHPC employs by adding delay.
> However 2 separate hotplug events as in your reproducer should be covered
> by the 1st patch.

Thank you very much for the explanation.

That indicates the two PCI devices will be detected and enabled in the same
event. Neither of the two PCI devices used to be enabled.

As mentioned in another email, I do not think this is the way to even workaround
the issue, because there are other ways to do mmio at the same time point.

Dongli Zhang

> 
>> Thank you very much!
>>
>> Dongli Zhang
>>
>>> HBA will start re-configuring bridge.
>>> Do the same thing as SHPC and throttle down hotplug of 2nd and up
>>> devices within single hotplug event.
>>>
>>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>> ---
>>>  drivers/pci/hotplug/acpiphp_glue.c | 6 ++++++
>>>  1 file changed, 6 insertions(+)
>>>
>>> diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c
>>> index 6b11609927d6..30bca2086b24 100644
>>> --- a/drivers/pci/hotplug/acpiphp_glue.c
>>> +++ b/drivers/pci/hotplug/acpiphp_glue.c
>>> @@ -37,6 +37,7 @@
>>>  #include <linux/mutex.h>
>>>  #include <linux/slab.h>
>>>  #include <linux/acpi.h>
>>> +#include <linux/delay.h>
>>>  
>>>  #include "../pci.h"
>>>  #include "acpiphp.h"
>>> @@ -700,6 +701,7 @@ static void trim_stale_devices(struct pci_dev *dev)
>>>  static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
>>>  {
>>>  	struct acpiphp_slot *slot;
>>> +        int nr_hp_slots = 0;
>>>  
>>>  	/* Bail out if the bridge is going away. */
>>>  	if (bridge->is_going_away)
>>> @@ -723,6 +725,10 @@ static void acpiphp_check_bridge(struct acpiphp_bridge *bridge)
>>>  
>>>  			/* configure all functions */
>>>  			if (slot->flags != SLOT_ENABLED) {
>>> +				if (nr_hp_slots)
>>> +					msleep(1000);
>>> +
>>> +                                ++nr_hp_slots;
>>>  				enable_slot(slot, true);
>>>  			}
>>>  		} else {  
>>
>