Received-SPF: pass (google.com: domain of linux-kernel+bounces-184895-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <afa06fba-43b4-4068-9639-ef4a6dfd3cef@oracle.com>
Date: Tue, 21 May 2024 07:47:49 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2] scsi: libsas: Fix exp-attached end device cannot be
 scanned in again after probe failed
To: yangxingui <yangxingui@huawei.com>, yanaijie@huawei.com,
        jejb@linux.ibm.com, martin.petersen@oracle.com,
        damien.lemoal@opensource.wdc.com
Cc: linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org,
        linuxarm@huawei.com, prime.zeng@hisilicon.com,
        chenxiang66@hisilicon.com, kangfenglong@huawei.com
References: <20240424080807.8469-1-yangxingui@huawei.com>
 <c1835d80-ca48-766e-c174-d94a2d357925@huawei.com>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <c1835d80-ca48-766e-c174-d94a2d357925@huawei.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?SWFkUFo4WW1ncUMybEExZHdlU0l6N2tNR0prckJnS3lEb3gwYk10U3FlL0pM?=
 =?utf-8?B?bnU1enFaRCtZYjZ5Nm5mSHdTYzJMekUxNmxmQnV0R3hiVjlHbHdxVDlJUHdY?=
 =?utf-8?B?M1MrTWRvQTVOUW5TZGRaRFhQempLUVhGVHdBb0ZEZVU4Tml1TFBSSmx4dzQ3?=
 =?utf-8?B?RktWYWEyVmdJNGs1MWZiVy9COVlQSkkzcDBrSUdsOTJ2dW0remlYeU8veERs?=
 =?utf-8?B?cmNYSTMyQzdMTjNqUE9QMHlQdTdVdGtFbURjUS8xTWlJRFQvZTZpaVZEQ2R1?=
 =?utf-8?B?M0tPSFZ3djBGM2syb3VUdTBaSE1tZkxrbHBqeVV0Ty9JYmovUjdJMjRhMjI3?=
 =?utf-8?B?Z1pYc0dQS01Dc1c0Wm5wNU4yU2dzRU9WcjU2QjNPM0l3b21hUWszS1dPNHFw?=
 =?utf-8?B?eGhiZndzeGFHditheERUY0dYeUdrSU1RQlRma1pPQ29lN1JjNGQrVXRkMDdp?=
 =?utf-8?B?akFpUXNEaUN3aFNZV2M3RkVndXV6Y3FWc0Z2MHNiL0NGeTBGaUZHQWpocG40?=
 =?utf-8?B?UUdEa1pzRzFTZmVNdi9VdzUwZCtnYWNyWHExNVI2VVNyRFpiSWthMXNzMmUr?=
 =?utf-8?B?YTRNYWJPNG1IY0NnVUFIaDNZWjFsS2Jxc2xacUJtK1JhVExnaFpKQWduY2kx?=
 =?utf-8?B?ZEErSHk3NDZSVWtYdkVEL25YMStBbU00KzFRckduTlpXS1lEY3VFMjBiQ0hw?=
 =?utf-8?B?ZjB3aWRIOGcxR0J6cmRDdGh2ME8xRTB2UkRCU1Z6RTlDaXNSb0dWczU2K2RO?=
 =?utf-8?B?ZTNlUnE4OElnMDMxN3FQa0doTzEwdUVZVVFFcWZxSTBHeDhYbHMrSEZUMlM3?=
 =?utf-8?B?QndDbU9lYkJmYlhKS2VTMDNCUyt1Vk54Zm1VRDl5NFBEaVg1NlIrQzk0RWZP?=
 =?utf-8?B?WUFLMExEREVuQndpM2djWldKOGNPZkNvNUlBZEs1UGl1elR6bGxYY3Zaa3Jk?=
 =?utf-8?B?MTdLZ2xjSDNkUkdUUnczNGhUeEt3eEh0MUp0MXN3RVRZRXVkR3JqLzhGb0JS?=
 =?utf-8?B?VlFSMGdhU3ZnTGRJd3dQbG9jRmNIb1BtbXlQVElnNDRnR3NUaXRKTmZ0STdS?=
 =?utf-8?B?cU0rcGJNZGIwL3JiQ0drRUkyUEFpc083bTJUNEd5c1BIRVBMT0hZZ0FjeTVj?=
 =?utf-8?B?WW5BeTBJTEVuZkxjNC9KMndDb2xlVHk3b1llR0lYMlRJZlVTcDRhdjVITmp6?=
 =?utf-8?B?NytyOHlUbEVxZUVxazQzTlFpOHM1TnI1UnEzc01oL0xnQlNVQmlNNGNyVkFn?=
 =?utf-8?B?ejNVZFE3Qnkra3FrUFkyN0RHY2tCSmd5eFhYQjhDRE5wRW5KV0p4c2ZuVjg5?=
 =?utf-8?B?Wis4R2NBaW1RZjVZNi8rSWF2RWtKNEcxN3NMa2Mrd2x3Zngwamd0NDBzRGJE?=
 =?utf-8?B?eklHUGZuZDJIemU1Y3BkLzIwaEVsc1dUVHVhS0RBRllDRnBkMDdSNUZTNktO?=
 =?utf-8?B?LzdUakl6K2RPZWNtVXlrYytKWEw1eXMwR3EwYVFZQWJ6cjZDUk0zUlI4WGRL?=
 =?utf-8?B?end5UElRbGx5SkJ2bUlINndaRWRUbGI0UlNlVXJhT3lzWmUzN0ROSmVnUHcx?=
 =?utf-8?B?T1FlU0x2SlhDNm93REpBMk1DOCtNeFlQOUlsdCtEanNoeWY2bUtIRWtVd2dx?=
 =?utf-8?B?eXZZTDROMUFsdXRUTVcwcWs2eG1TM0hpbzVwRXhCWXZkaFRuZXBpQW1JNTls?=
 =?utf-8?B?R2YyUGFZclFJSm44MENOTXBxSm04M3pIbDI4RWVURElZa2tJUWdnS1pVblRT?=
 =?utf-8?B?SmQ1c09iUWlDSXRDMjJ2RTB1cUJFclcxVVRQOFJiM2YzQ29uYWhxeEJKRk5E?=
 =?utf-8?B?eWx4S3Q3QXdVUXpOWjdlZlI0WncvWWFZZ0VnZVZBU0ZOK2xVQUpOUUtVeXE5?=
 =?utf-8?B?a0dabWtac080N2wwSHZGOExHeGZUVkNVOUF3d2pVZUgweVJ0NEw4ano2Qy9G?=
 =?utf-8?B?RGVlcFpJYWs5VDhIMURJUWRtRjJmejJ1eHkzcEJKQy8vVnR1YnFBYnNHOHU3?=
 =?utf-8?B?Nms1c05aRDdSKy80U1Q5bGJEOFdWWTNoUkdZeEJhVUQ5ejRxMWMxK0FNd3c3?=
 =?utf-8?B?UHlIdS8rRjN5dWYwU0dsL3ZVTDNTL3lhYXRpWGVsWTNHOFJkcVRrVUJhamV0?=
 =?utf-8?B?cWNDZ3BNcU93b0dmWDZKL3J4NnRqbm1qYjZzZ1RCMm0wakJLemxLMk9kdlBJ?=
 =?utf-8?B?elE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	m5qNL/glN4BXW8vmzzqvjllBY3CqYkq58dbPrkz4Zerolj3NwR7xKJTgn5VzViz9ngWx1r9KbvObMWAZICRT5OFzZkUeFU4mxFgkMTcFtjgidoFOiPr017MJfSti9cCRyYHVXYmHNLLWza+sn6+H+D80i3p+ehZKE2EuOcTYCIYBOa8Kusbheq6FaM2pmdpSEj9t2MtI+ISVwGH8YN+Imfra8uiTn7CAQ7ISXYN/FtfMX6PjyrMk2V9NvkpywRpI7UCyMB6Hg0jRwXRG9tYxPkd45VXd+KzCVpgfOsepQnBXH7XIYroLHsl/i7x8ihK4sfd67Mj4XCSmR3iOH5vDHfj+5avPpSjQJvSLWNx3rd9zqPwd5O+cCVhLqwOQVRcqDi46vawy6NPsbg+1TKV1g5B5mV9nLxWXEw/Tav/XuTo1UQYUhKE6mI6QrMYVVfeca29gsJBblFL1ic24nDMTYMF25x4JUrsZjye4q6Xdhaexx0hmnGivZQQEJ25pvP7ZttYvS/LiL6IWIjo01A82grkGnbtRT/AbjvAvlwfZhxpIx25VRZM4FxRb53Io32JQnSM63UhGob1LXJbGA0/+qmNS+PqUyDl/6gSstGQbXQk=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cab91329-8fb0-4d0f-1d33-08dc798bd8ab
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2024 11:47:52.3418
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HiINGvyEso3uq6IJQKsfBaXXmE0gnSNl7kOL2J3ZK35FY22QU9pRlRoNpr4HZXr/nTcCi99B/nyDBtE1z+o+HQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7899
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-21_07,2024-05-21_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 bulkscore=0
 suspectscore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405210088
X-Proofpoint-ORIG-GUID: S1flwL1-z1w2J7rQNp9wQU3COzZT8yhq
X-Proofpoint-GUID: S1flwL1-z1w2J7rQNp9wQU3COzZT8yhq

On 20/05/2024 09:29, yangxingui wrote:
> Friendly ping ...

I'll check when I return from vacation in a couple of days

> 
> On 2024/4/24 16:08, Xingui Yang wrote:
>> We found that it is judged as broadcast flutter when the exp-attached end
>> device reconnects after probe failed, as follows:
>>
>> [78779.654026] sas: broadcast received: 0
>> [78779.654037] sas: REVALIDATING DOMAIN on port 0, pid:10
>> [78779.654680] sas: ex 500e004aaaaaaa1f phy05 change count has changed
>> [78779.662977] sas: ex 500e004aaaaaaa1f phy05 originated 
>> BROADCAST(CHANGE)
>> [78779.662986] sas: ex 500e004aaaaaaa1f phy05 new device attached
>> [78779.663079] sas: ex 500e004aaaaaaa1f phy05:U:8 attached: 
>> 500e004aaaaaaa05 (stp)
>> [78779.693542] hisi_sas_v3_hw 0000:b4:02.0: dev[16:5] found
>> [78779.701155] sas: done REVALIDATING DOMAIN on port 0, pid:10, res 0x0
>> [78779.707864] sas: Enter sas_scsi_recover_host busy: 0 failed: 0
>> ...
>> [78835.161307] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 
>> tries: 1
>> [78835.171344] sas: sas_probe_sata: for exp-attached device 
>> 500e004aaaaaaa05 returned -19
>> [78835.180879] hisi_sas_v3_hw 0000:b4:02.0: dev[16:5] is gone
>> [78835.187487] sas: broadcast received: 0
>> [78835.187504] sas: REVALIDATING DOMAIN on port 0, pid:10
>> [78835.188263] sas: ex 500e004aaaaaaa1f phy05 change count has changed
>> [78835.195870] sas: ex 500e004aaaaaaa1f phy05 originated 
>> BROADCAST(CHANGE)
>> [78835.195875] sas: ex 500e004aaaaaaa1f rediscovering phy05
>> [78835.196022] sas: ex 500e004aaaaaaa1f phy05:U:A attached: 
>> 500e004aaaaaaa05 (stp)
>> [78835.196026] sas: ex 500e004aaaaaaa1f phy05 broadcast flutter
>> [78835.197615] sas: done REVALIDATING DOMAIN on port 0, pid:10, res 0x0
>>
>> The cause of the problem is that the related ex_phy's 
>> attached_sas_addr was
>> not cleared after the end device probe failed. In order to solve the 
>> above
>> problem, a function sas_ex_unregister_end_dev() is defined to clear the
>> ex_phy information and unregister the end device after the 
>> exp-attached end
>> device probe failed.
>>
>> As the sata device is an asynchronous probe, the sata device may probe

SATA, almost always capitalize acronyms

>> failed after done REVALIDATING DOMAIN. Then after its port is added to 
>> the
>> sas_port_del_list, the port will not be deleted until the end of the next
>> REVALIDATING DOMAIN and sas_destruct_ports() is called. A warning about
>> creating a duplicate port will occur in the new REVALIDATING DOMAIN when
>> the end device reconnects. Therefore, the previous destroy_list and
>> sas_port_del_list should be handled before REVALIDATING DOMAIN.
>>
>> Signed-off-by: Xingui Yang <yangxingui@huawei.com>
>> ---
>> Changes since v1:
>> - Simplify the process of getting ex_phy id based on Jason's suggestion.
>> - Update commit information.
>> ---
>>   drivers/scsi/libsas/sas_discover.c | 2 ++
>>   drivers/scsi/libsas/sas_expander.c | 8 ++++++++
>>   drivers/scsi/libsas/sas_internal.h | 6 +++++-
>>   3 files changed, 15 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/scsi/libsas/sas_discover.c 
>> b/drivers/scsi/libsas/sas_discover.c
>> index 8fb7c41c0962..aae90153f4c6 100644
>> --- a/drivers/scsi/libsas/sas_discover.c
>> +++ b/drivers/scsi/libsas/sas_discover.c
>> @@ -517,6 +517,8 @@ static void sas_revalidate_domain(struct 
>> work_struct *work)
>>       struct sas_ha_struct *ha = port->ha;
>>       struct domain_device *ddev = port->port_dev;
>> +    sas_destruct_devices(port);
>> +    sas_destruct_ports(port);
>>       /* prevent revalidation from finding sata links in recovery */
>>       mutex_lock(&ha->disco_mutex);
>>       if (test_bit(SAS_HA_ATA_EH_ACTIVE, &ha->state)) {
>> diff --git a/drivers/scsi/libsas/sas_expander.c 
>> b/drivers/scsi/libsas/sas_expander.c
>> index f6e6db8b8aba..45793c10009b 100644
>> --- a/drivers/scsi/libsas/sas_expander.c
>> +++ b/drivers/scsi/libsas/sas_expander.c
>> @@ -1856,6 +1856,14 @@ static void sas_unregister_devs_sas_addr(struct 
>> domain_device *parent,
>>       }
>>   }
>> +void sas_ex_unregister_end_dev(struct domain_device *dev)
>> +{
>> +    struct domain_device *parent = dev->parent;
>> +    struct sas_phy *phy = dev->phy;
>> +
>> +    sas_unregister_devs_sas_addr(parent, phy->number, true);
>> +}
>> +
>>   static int sas_discover_bfs_by_root_level(struct domain_device *root,
>>                         const int level)
>>   {
>> diff --git a/drivers/scsi/libsas/sas_internal.h 
>> b/drivers/scsi/libsas/sas_internal.h
>> index 3804aef165ad..434f928c2ed8 100644
>> --- a/drivers/scsi/libsas/sas_internal.h
>> +++ b/drivers/scsi/libsas/sas_internal.h
>> @@ -50,6 +50,7 @@ void sas_discover_event(struct asd_sas_port *port, 
>> enum discover_event ev);
>>   void sas_init_dev(struct domain_device *dev);
>>   void sas_unregister_dev(struct asd_sas_port *port, struct 
>> domain_device *dev);
>> +void sas_ex_unregister_end_dev(struct domain_device *dev);
>>   void sas_scsi_recover_host(struct Scsi_Host *shost);
>> @@ -145,7 +146,10 @@ static inline void sas_fail_probe(struct 
>> domain_device *dev, const char *func, i
>>           func, dev->parent ? "exp-attached" :
>>           "direct-attached",
>>           SAS_ADDR(dev->sas_addr), err);
>> -    sas_unregister_dev(dev->port, dev);
>> +    if (dev->parent && !dev_is_expander(dev->dev_type))
>> +        sas_ex_unregister_end_dev(dev);
>> +    else
>> +        sas_unregister_dev(dev->port, dev);
>>   }
>>   static inline void sas_fill_in_rphy(struct domain_device *dev,
>>