Received-SPF: pass (google.com: domain of linux-kernel+bounces-188472-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <824c34aa-7c4c-4edd-b41c-f9b5ff5aff03@oracle.com>
Date: Fri, 24 May 2024 09:36:38 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2] scsi: libsas: Fix exp-attached end device cannot be
 scanned in again after probe failed
To: Xingui Yang <yangxingui@huawei.com>, yanaijie@huawei.com,
        jejb@linux.ibm.com, martin.petersen@oracle.com,
        damien.lemoal@opensource.wdc.com
Cc: linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org,
        linuxarm@huawei.com, prime.zeng@hisilicon.com,
        chenxiang66@hisilicon.com, kangfenglong@huawei.com
References: <20240424080807.8469-1-yangxingui@huawei.com>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <20240424080807.8469-1-yangxingui@huawei.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?RVJLR1oydHgyblhHeFpSOXFQbDl4OVZzK0NmZ0RPczgyaklmZVVMS1VtN1Aw?=
 =?utf-8?B?VXJoalZFc0EwMXJHM1l2RjQ1a0wrak8yLzJWUzZNNW1zbUg4NC8xdmFDemgy?=
 =?utf-8?B?TmxiQUFQYXoxbmpaK0lqc0hObzNHaEZVZjhSMUhzb09IVXhCQ1dZSzZCVnZZ?=
 =?utf-8?B?N2dTaHlPdTIxb1NqSEVCWGdHOFQ1bkVpRFZMVC9mM0NDYnZYdTFJVmpSRjN2?=
 =?utf-8?B?MnQ0RS81cFdDWEZvL01uMUNNUTdNRWIxakZGVDdXTERQYTVCRldlK3ZFOG9T?=
 =?utf-8?B?bUduWUp5bDlydHd2NHdvYVh6Y0JTdjc2WW45UERUUU8yd2VkTVVsL1M2S1g1?=
 =?utf-8?B?aW5ubTE0ZDl4dmR2cVFhakdFVnJtWXJsSkJ6MVROTXp1RDhOT0VwVUZXU0d1?=
 =?utf-8?B?UlRQSUVjbGlHTDNuSEkrSmVJQnpxdHlQa1NJcCt3QStYT2FOa0VPYnMyT3pW?=
 =?utf-8?B?clM5eXBNSFEzTmt6bVZ4cStoa2VXRjZNNjFLM3pubkgzL081ckRGZjRGeXc0?=
 =?utf-8?B?L1g2UGZMOVJiTXhDZm04bXFmZFRsclRHZmNUNHVxQ2xuMTl4a0ZDUWt6ZnNj?=
 =?utf-8?B?MkdheHRCK3Q2cmY4eHdQODBpYVhWTkx0TWFaTmZtck1ORzhlZzhRMkUrdE1L?=
 =?utf-8?B?MERFdCs1VERPT2llSnJ0bkNRNVVLVnJsYzBaUUdCUzBVRDllRkowTWVvMUlE?=
 =?utf-8?B?ejV4MmIvSUNDZ2pjM3ZqTDVaMkJydTFmckF1NCtrT3BkYVFVQ2xTTU9ldEdQ?=
 =?utf-8?B?cWh1Yno0WUhtQktUcmRRZDRtR3kwKzJVT2gwNTJpV21CbDkwVXBQSTNaZG1n?=
 =?utf-8?B?WkJmOHdlbEVITk5ZNlVrRmdTUmliaStPUzZHMUJhREJGL2NBVExIck9SSFNL?=
 =?utf-8?B?Zy9YUXJjak9rMEZyVkV5a3FaaVE1MjR3WmtxR0t4K2JwcTBYM0tUQTZhbHNr?=
 =?utf-8?B?a2l4YXIrMURHTWhRVkozRjZweWk0Q0FHRS9HdlhLdXZCRVNyKzlqcnllZlpK?=
 =?utf-8?B?MUFFakw0UC85OEgxNlJDTzU0RzhaY3ZYaEZ2VWlDWDNVK0srNHRjQnYrWWJK?=
 =?utf-8?B?Y0hKS2t5c2RpSGRFamh0Q1g5TjFBL2ZaYWF4WUdabFVPa0o0a1h1WExHSXlN?=
 =?utf-8?B?WmorMjFhQWFGa2hDeDcyUTR5aTAxT3U1SDZiS2RBazhYRVNhSVpsZDE5WXpr?=
 =?utf-8?B?QVl3c0RqTTRuNldTeVFqWXBJNVZLeHdoY01waTdGMi9uZmd1V0QxcUttS2lo?=
 =?utf-8?B?Ly8zTHNVZjJwV01FdHhwT3VpUkw2eHBxcDJ2UjQ0T2xjWU5JMFY3UTR4Rktz?=
 =?utf-8?B?MThuVWxMUTN4WVhFZWM4ZHp2MGlIcDB0UjY3V3lMTis0Y1l6UlVtVmVCUVpq?=
 =?utf-8?B?cmRxNmFlM0hFdlZEamNicnVBbisvOWxaTDd2YXJQRmozRll4VzFhc1U5WXRt?=
 =?utf-8?B?REVnb0diODlyWCs2N09yQThZZk1xY3hyQ0cxWkd5WkJxOHhZM1NCVVBSSHdD?=
 =?utf-8?B?ZjcrcEJNYU1DMkVUSnVXWGxmUTNyL2E3S0ZsVWZsQ3BrNWVLYklWd0pYY2Rq?=
 =?utf-8?B?Q1k5RkhHUy84RGRVbTc4ZUdmSi9OUWlSTDBiaEs2b1gzOURYV09UeWJjeis3?=
 =?utf-8?B?MGVvUUpCd0d4bEEzcSs1c3hTMno3RXh2Z2cxeVZFRktwVjZoME85MWRkMVNG?=
 =?utf-8?B?ZlNrVWJ4SUxUai9VNlpGcHljVDUzbW9tWVptSzhGMVFQR0tSWi8rbEZJRFpO?=
 =?utf-8?B?OUYwcC92c2xuKy80eTY0OXVTS2dEbzlseEZSalN0U3JiOU5SUWhIbjY0ZW5k?=
 =?utf-8?B?UTdJNEE5M0NheWdoWkEwbzhWMWFFb2VvMDJQV3JDSXFyaW5zb0FjWHNkcEh2?=
 =?utf-8?B?UENDa0grM1JkWnlXOHFOcEExZ2JFUjk3L1AzRE9YMDY4RnV5QllZZFN3U2hr?=
 =?utf-8?B?K045emVDNlpkdzNDUEd5bVkxMjYyckc3cCtjYVZ2R1hOUWhERVVFYmRvblBh?=
 =?utf-8?B?dGRFeS8yY2FvOGFSeUVwMFlOTGdkdDNJVGprVWFaZ053N1NaYmNXc2lDdGkv?=
 =?utf-8?B?YkRuUTB1UmoyRE9GZzQ5bGJ6aXpkU2liY2o5RWFQQ3FCNzI4REJVMCtJaU9j?=
 =?utf-8?B?RWxQbGJZZFErYzJPWGZZRGpIZEVLcWV2RHlBbEkyK3RSeU1RM1FteEUwRjAr?=
 =?utf-8?B?Wmc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	0nYDaGlRRAtk8PN9NQKMNQCwcXn6fULpLxGhm6nc43KNOmsZ3Yrf6xMeOpw+SKeR8kd6EVx5Ol1B+r06BvUKWdYaMXPHsA/0RlCqfqYjV7BiQ5orG2aqql/o82pZK4SbcuN9PBCJdZcpq5gl1iR0bQZk8jEVCd4X3v5MYnVlGQXTAu+wL7fuDbLypo1UXmI6bv7c4vGrPJvpYOAcAswZDvtldS3EOhZ/SUgn3jVDN0cw/AuvIPPTM5OIxV/mbv7fD86USy9FxTgJm8BCCRl/DJu34Er9UzvcsjlBMxFmT8EMSvNnolYNp5G7Un9CARPU92pbqjd69azERk1x4ScxYcT1OaZRnQaZL3EUR9wn3ZfRBVYCC+b8P6QUGHPPhklVIdqWyX19GUy73ZC6K0UfZakiT16fhOvG8ZQGGnZT1J4E1TTqU+FTH5sb3R56fMDodwbAyOIR4cUQGRZlzKILnFQf4+eRKkhAf8g2Bp4MBhHcRJN6n2zRVDLW1iYZFWafsM0TYOkVrsZe0ChO/shsFvJb3gQDfREcOjAu96O00R7xpayh7Z3l9iwED5Y3mAao3dkJpT72OzwAE3YhsGEyZmQsYnsCUNfGREDu7j5B2DY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 60ac42dc-737e-4f45-4e42-08dc7bcca51f
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2024 08:36:45.5702
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7shxQEaPfLkTXJ0wx5w8ViSPXyzytjpluss9Sa24TcIoP/LFHe//kYH5HNNkD7VtM66pwC5hBOAoTxu68rVkBA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4636
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-24_02,2024-05-23_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 phishscore=0
 suspectscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405240058
X-Proofpoint-ORIG-GUID: ao8bZh7VLen_vNbSHM2xzWmbFZTRZRad
X-Proofpoint-GUID: ao8bZh7VLen_vNbSHM2xzWmbFZTRZRad

On 24/04/2024 09:08, Xingui Yang wrote:
> We found that it is judged as broadcast flutter when the exp-attached end
> device reconnects after probe failed, as follows:
> 
> [78779.654026] sas: broadcast received: 0
> [78779.654037] sas: REVALIDATING DOMAIN on port 0, pid:10
> [78779.654680] sas: ex 500e004aaaaaaa1f phy05 change count has changed
> [78779.662977] sas: ex 500e004aaaaaaa1f phy05 originated BROADCAST(CHANGE)
> [78779.662986] sas: ex 500e004aaaaaaa1f phy05 new device attached
> [78779.663079] sas: ex 500e004aaaaaaa1f phy05:U:8 attached: 500e004aaaaaaa05 (stp)
> [78779.693542] hisi_sas_v3_hw 0000:b4:02.0: dev[16:5] found
> [78779.701155] sas: done REVALIDATING DOMAIN on port 0, pid:10, res 0x0
> [78779.707864] sas: Enter sas_scsi_recover_host busy: 0 failed: 0
> ...
> [78835.161307] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1
> [78835.171344] sas: sas_probe_sata: for exp-attached device 500e004aaaaaaa05 returned -19
> [78835.180879] hisi_sas_v3_hw 0000:b4:02.0: dev[16:5] is gone
> [78835.187487] sas: broadcast received: 0
> [78835.187504] sas: REVALIDATING DOMAIN on port 0, pid:10
> [78835.188263] sas: ex 500e004aaaaaaa1f phy05 change count has changed
> [78835.195870] sas: ex 500e004aaaaaaa1f phy05 originated BROADCAST(CHANGE)
> [78835.195875] sas: ex 500e004aaaaaaa1f rediscovering phy05
> [78835.196022] sas: ex 500e004aaaaaaa1f phy05:U:A attached: 500e004aaaaaaa05 (stp)
> [78835.196026] sas: ex 500e004aaaaaaa1f phy05 broadcast flutter
> [78835.197615] sas: done REVALIDATING DOMAIN on port 0, pid:10, res 0x0
> 
> The cause of the problem is that the related ex_phy's attached_sas_addr was
> not cleared after the end device probe failed. In order to solve the above
> problem, a function sas_ex_unregister_end_dev() is defined to clear the
> ex_phy information and unregister the end device after the exp-attached end
> device probe failed.
> 
> As the sata device is an asynchronous probe, the sata device may probe
> failed after done REVALIDATING DOMAIN. Then after its port is added to the
> sas_port_del_list, the port will not be deleted until the end of the next
> REVALIDATING DOMAIN and sas_destruct_ports() is called. A warning about
> creating a duplicate port will occur in the new REVALIDATING DOMAIN when
> the end device reconnects. Therefore, the previous destroy_list and
> sas_port_del_list should be handled before REVALIDATING DOMAIN.
> 
> Signed-off-by: Xingui Yang <yangxingui@huawei.com>
> ---
> Changes since v1:
> - Simplify the process of getting ex_phy id based on Jason's suggestion.
> - Update commit information.
> ---
>   drivers/scsi/libsas/sas_discover.c | 2 ++
>   drivers/scsi/libsas/sas_expander.c | 8 ++++++++
>   drivers/scsi/libsas/sas_internal.h | 6 +++++-
>   3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/libsas/sas_discover.c b/drivers/scsi/libsas/sas_discover.c
> index 8fb7c41c0962..aae90153f4c6 100644
> --- a/drivers/scsi/libsas/sas_discover.c
> +++ b/drivers/scsi/libsas/sas_discover.c
> @@ -517,6 +517,8 @@ static void sas_revalidate_domain(struct work_struct *work)
>   	struct sas_ha_struct *ha = port->ha;
>   	struct domain_device *ddev = port->port_dev;
>   
> +	sas_destruct_devices(port);
> +	sas_destruct_ports(port);

We still have both these same calls at the @out label - is that as desired?

Why do these new additions not cover the same job which those calls to 
the same functions @out covers?

>   	/* prevent revalidation from finding sata links in recovery */
>   	mutex_lock(&ha->disco_mutex);
>   	if (test_bit(SAS_HA_ATA_EH_ACTIVE, &ha->state)) {
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index f6e6db8b8aba..45793c10009b 100644
> --- a/drivers/scsi/libsas/sas_expander.c
> +++ b/drivers/scsi/libsas/sas_expander.c
> @@ -1856,6 +1856,14 @@ static void sas_unregister_devs_sas_addr(struct domain_device *parent,
>   	}
>   }
>   
> +void sas_ex_unregister_end_dev(struct domain_device *dev)
> +{
> +	struct domain_device *parent = dev->parent;
> +	struct sas_phy *phy = dev->phy;
> +
> +	sas_unregister_devs_sas_addr(parent, phy->number, true);
> +}
> +
>   static int sas_discover_bfs_by_root_level(struct domain_device *root,
>   					  const int level)
>   {
> diff --git a/drivers/scsi/libsas/sas_internal.h b/drivers/scsi/libsas/sas_internal.h
> index 3804aef165ad..434f928c2ed8 100644
> --- a/drivers/scsi/libsas/sas_internal.h
> +++ b/drivers/scsi/libsas/sas_internal.h
> @@ -50,6 +50,7 @@ void sas_discover_event(struct asd_sas_port *port, enum discover_event ev);
>   
>   void sas_init_dev(struct domain_device *dev);
>   void sas_unregister_dev(struct asd_sas_port *port, struct domain_device *dev);
> +void sas_ex_unregister_end_dev(struct domain_device *dev);
>   
>   void sas_scsi_recover_host(struct Scsi_Host *shost);
>   
> @@ -145,7 +146,10 @@ static inline void sas_fail_probe(struct domain_device *dev, const char *func, i
>   		func, dev->parent ? "exp-attached" :
>   		"direct-attached",
>   		SAS_ADDR(dev->sas_addr), err);
> -	sas_unregister_dev(dev->port, dev);
> +	if (dev->parent && !dev_is_expander(dev->dev_type))

This check looks odd.

So we're checking if we have a parent device and we are not an expander, 
right?

> +		sas_ex_unregister_end_dev(dev);
> +	else
> +		sas_unregister_dev(dev->port, dev);
>   }
>   
>   static inline void sas_fill_in_rphy(struct domain_device *dev,