Received-SPF: pass (google.com: domain of linux-kernel+bounces-203299-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <fec6ba82-aef8-4ffe-a18b-20ac8e0a1a03@os.amperecomputing.com>
Date: Wed, 5 Jun 2024 14:32:55 -0700
User-Agent: Mozilla Thunderbird
From: Daniel Ferguson <danielf@os.amperecomputing.com>
Subject: Re: [RFC PATCH v8 08/10] ACPI:RAS2: Add ACPI RAS2 driver
To: shiju.jose@huawei.com
Cc: ira.weiny@intel.com, vishal.l.verma@intel.com,
 alison.schofield@intel.com, dave.jiang@intel.com,
 jonathan.cameron@huawei.com, dave@stgolabs.net, dan.j.williams@intel.com,
 linux-mm@kvack.org, linux-acpi@vger.kernel.org, linux-cxl@vger.kernel.org,
 linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org, david@redhat.com,
 Vilas.Sridharan@amd.com, leo.duran@amd.com, Yazen.Ghannam@amd.com,
 rientjes@google.com, jiaqiyan@google.com, tony.luck@intel.com,
 Jon.Grimm@amd.com, dave.hansen@linux.intel.com, rafael@kernel.org,
 lenb@kernel.org, naoya.horiguchi@nec.com, james.morse@arm.com,
 jthoughton@google.com, somasundaram.a@hpe.com, erdemaktas@google.com,
 pgonda@google.com, duenwen@google.com, mike.malvestuto@intel.com,
 gthelen@google.com, wschwartz@amperecomputing.com,
 dferguson@amperecomputing.com, wbs@os.amperecomputing.com,
 nifan.cxl@gmail.com, tanxiaofei@huawei.com, prime.zeng@hisilicon.com,
 kangkang.shen@futurewei.com, wanghuiqiang@huawei.com, linuxarm@huawei.com
References: <20240419164720.1765-1-shiju.jose@huawei.com>
 <20240419164720.1765-9-shiju.jose@huawei.com>
Content-Language: en-US
In-Reply-To: <20240419164720.1765-9-shiju.jose@huawei.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WEpMRXFBOSt6bnlHa0o5eTBhVnZkeU90a0VrUk4xRjhRTFdMR2NrTmJjeW1q?=
 =?utf-8?B?cm40N0VWZ3htWnRGUFNMOUt1anZXWjAwY3B2QmRzRmFJSER1QkJlRUpTMFdr?=
 =?utf-8?B?SmdBOEhHenJxQUxzSjNhVmlWRHJaeUR1Z1JPYTQzRmJWWlRBRWtWakFtM2ZJ?=
 =?utf-8?B?d0F0UmZ2MFZYVEdDZjFWTUppb0R0VURRTUZiU2lvMlRiTTltMEtrY3hqeXdu?=
 =?utf-8?B?Y2E4VUY4MXhtdjBXTE10WGxva1Zub0xvelBZcDZaOVV6ckZjZVZXU2lpVk1R?=
 =?utf-8?B?T3BuZmpYUFN6bE1nalVrcSs2WDhobGErakw3NnVBbXFVNThCRU1DODgzYnFO?=
 =?utf-8?B?Y2FFaUJvMm1IbkN1MTY3bGc3YUwxajVYRHdsSGF5YUJIbnlidGcxb1ZTOEY4?=
 =?utf-8?B?dHcrb3ZHK0wxTGtXWjVKbnFzVzIwZm43Tm93bHV6UHpXUDd1c1BtODZWbnp3?=
 =?utf-8?B?bVVkY0RDRW1QMG96TlFHNUtTeFJpTnBHTE84bVQyQjRDZUdVZndzcXRMaHpk?=
 =?utf-8?B?WW9BZ0N2UlY5anExT0FXZnluRHlzYmxFWW9vNXpXUEZOQS8xajZ3enp1dFht?=
 =?utf-8?B?K1YzMVN6WFYwM2pwckxIRnRlM2p6THBPK3NTZ0twNnU0UXFId3ovc1pVbmNm?=
 =?utf-8?B?S01lS0VOdW9XTG92UjRSb3QxKzJPL1BPdlRDN1NXRmVhdTFKTWRqWFhwWFBp?=
 =?utf-8?B?UjlBT3p2Y2ZNNm1SUENkL0RMNlRDaGdGQ3EzaGFxVTJlTkJ1WGtNRTQzOEV0?=
 =?utf-8?B?ckNrbFN5MHVadktZT2NrZFkzZlVGZ1h1Mk9KRTZQLzVBdjRpbFprVGJHQTdX?=
 =?utf-8?B?amd5Q0NFTHhsK3ZhTk9qYWZDekVoRkV1Mjk2YXRZUWE2cjFXZGpsRUhDbG5o?=
 =?utf-8?B?SXRaWkUrUXZYMnduVGtqejRlK0N4SzcvMlEyT0NSN3NPcmp1a2NqS2VrRnB2?=
 =?utf-8?B?OExTQWhJRkUrN2tUc0tlUmQ0SStUYmtDbHdqbUE5V01KUU05ODRqeEZ5Zk5i?=
 =?utf-8?B?YnAzYmZLQWlJc2xDVUFTVFk4ZHFYb2g5cGVpaWIxdGpYZ0xSN1N4OVZWOVFn?=
 =?utf-8?B?R05KVENrOWZZN3hQTUlyTVJvazBSVGcyRlQyTHhGTEF2UlN3U2M2V3Jtem00?=
 =?utf-8?B?ODdSUGVxK0M5VzgrRXlrR3d0ajJsVVJCZW94bWx2UElsalBmY3RmRllPTCt3?=
 =?utf-8?B?ZW4vUFZTVEQwcm9RRnNMSUF2S2x2Q0dxbEtZWlFNU0RDZERSQVdHdU5IdXd4?=
 =?utf-8?B?azhmVDlSaVZSdmNXVkFpNDI1ZFRGbDNZcUNUZGxhYlBlQ2pUaS9QdVE4U1pW?=
 =?utf-8?B?Y01iYUh0dnFodTE3TW8yMFpTTk10Y1RiOEY4RkM4RnBWdFBTekdCRzNuZi9z?=
 =?utf-8?B?L3VTTTBqQU9jdTFaMlVVdEo4azlLY1BvWjNCSEpJU2ZLS0R5dWlwd0xGY3Jx?=
 =?utf-8?B?YXMzdWV4M2laMUd5dFRTQ3c1aC90ZW9TZkVVQTNLT2dxeVZYNWlBQnpyVTFm?=
 =?utf-8?B?VW92bit2VWk5NUFoL0xhKzkva0JiYVBRQkhTaUp5MnJaUUQ2d3VGMzVybExh?=
 =?utf-8?B?U00xSXpoaEppWWd1bmM4bldEaDJXeTNzaFFHRzdQbW40eDJsejkvRTFUMzJ5?=
 =?utf-8?B?UmVpb2JFSTNzRVpWdkc2N1paSHdWbFJyNmkzUlFJeXVSWkwydVRuem42TDNm?=
 =?utf-8?B?WFBxb0o2YW8rbnEzMmY1YmxOWm1DUjVheCtHd3lRYVNxZUowTGhRajhHaWlk?=
 =?utf-8?B?TGRVeUhiUVoyNGtnU2NRRnplazd5bHVidE5WNFMvRVFvNXk5OGRDOEtkUXVi?=
 =?utf-8?B?R0xDZTBCazN4bitSa1VIeHdMczVHNllCV0p4dDVVcDJDSTBqbmtOa2xDZFJv?=
 =?utf-8?B?SXNGUk1VZXd2NUNXQkJQaUNNcDJHaHhXa3RmWXF6R0FYWVQxSTVVQVExdUl0?=
 =?utf-8?B?bGtNTkxORmdaQlpMU2xzM2xONTI0ajlJRlJaL2ttQWV6ZTdvNnozN282QVpt?=
 =?utf-8?B?T0cvdzFDZEd3RUVjMHFoc0V2NVVhTHZKTnZvakhJdjRYaW0zUnFjZzVwTVFE?=
 =?utf-8?B?OUJhMFhTZGpFb1VKZ1o5MjZSL05GZ2JoNFRoZzBydUdBbmU2OU84dHZ1L2U0?=
 =?utf-8?B?ak02aE0zdVJqdjBlV3V1UHdjNzhPT005U3pDVU9pNGV5ancxREw5OEJlbFds?=
 =?utf-8?Q?PeP2crEHQmXFDlDy2402tIg=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 60a3abb0-2605-4905-1787-08dc85a71390
X-MS-Exchange-CrossTenant-AuthSource: CO1PR01MB7370.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2024 21:33:01.6331
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KuAsfeDpHsllQ/s4ueeeor3nWxoMwljsZ4Svx7co1fF3AEDhvh4SxVuRsxIuwiuYLPaxtVtoFg+QlcSEk1OrNmfbxg3mNTlCrfCq5t+zA/MzGY2qSpKaOIjfjs+NbiN6
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR01MB8462

> +static int ras2_check_pcc_chan(struct ras2_pcc_subspace *pcc_subspace)
> +{
> +	struct acpi_ras2_shared_memory __iomem *generic_comm_base =pcc_subspace->pcc_comm_addr;
> +	ktime_t next_deadline = ktime_add(ktime_get(), pcc_subspace->deadline);
> +	u16 status;
> +
> +	while (!ktime_after(ktime_get(), next_deadline)) {
> +		/*
> +		 * As per ACPI spec, the PCC space will be initialized by
> +		 * platform and should have set the command completion bit when
> +		 * PCC can be used by OSPM
> +		 */
> +		status = readw_relaxed(&generic_comm_base->status);
> +		if (status & RAS2_PCC_CMD_ERROR)
> +			return -EIO;

We need to clear the error bit before reporting an error.
Maybe the error was transient, or specific to the last transaction.
So clearing it here, lets us try again later. Like This:
		if (status & RAS2_PCC_CMD_ERROR) {
			status &= ~RAS2_PCC_CMD_ERROR;
			writew_relaxed(status, &generic_comm_base->status);
			return -EIO;
		}


Also, we are thinking that using the "Set RAS Capability Status" as a
way to communicate RAS2 Scrub specific error conditions is appropriate.
If we agree on that idea, then we can additionally check the
set_capabilities_status and return an appropriate error..
for example, we can add a new error mapping function:
static int report_cap_error(u32 cap_status)
{
	switch (cap_status) {
		case 1:  /* Not Valid */
		case 2:  /* Not Supported */
			return -EPERM;
		case 3:  /* Busy */
			return -EBUSY;
		case 4:  /* FailedF */
		case 5:  /* Aborted */
		case 6:  /* Invalid Data */
			return -EINVAL;
		default: /* 0 or other, Success */
			return 0;
	}
}

and then instead, modify ras2_check_pcc_chan in this way:
		if (status & RAS2_PCC_CMD_ERROR) {
			cap_status = readw_relaxed(&generic_comm_base->set_capabilities_status);
			ret = report_cap_error(cap_status);

			status &= ~RAS2_PCC_CMD_ERROR;
			writew_relaxed(status, &generic_comm_base->status);
			return ret;
		}

> +		if (status & RAS2_PCC_CMD_COMPLETE)
> +			return 0;
> +		/*
> +		 * Reducing the bus traffic in case this loop takes longer than
> +		 * a few retries.
> +		 */
> +		msleep(10);
> +	}
> +
> +	return -EIO;
> +}
> +
> +/**
> + * ras2_send_pcc_cmd() - Send RAS2 command via PCC channel
> + * @ras2_ctx:	pointer to the ras2 context structure
> + * @cmd:	command to send
> + *
> + * Returns: 0 on success, an error otherwise
> + */
> +int ras2_send_pcc_cmd(struct ras2_scrub_ctx *ras2_ctx, u16 cmd)
> +{
> +	struct ras2_pcc_subspace *pcc_subspace = ras2_ctx->pcc_subspace;
> +	struct acpi_ras2_shared_memory *generic_comm_base = pcc_subspace->pcc_comm_addr;
> +	static ktime_t last_cmd_cmpl_time, last_mpar_reset;
> +	struct mbox_chan *pcc_channel;
> +	unsigned int time_delta;
> +	static int mpar_count;
> +	int ret;
> +
> +	guard(mutex)(&ras2_pcc_subspace_lock);
> +	ret = ras2_check_pcc_chan(pcc_subspace);
> +	if (ret)
> +		return ret;
> +	pcc_channel = pcc_subspace->pcc_chan->mchan;
> +
> +	/*
> +	 * Handle the Minimum Request Turnaround Time(MRTT)
> +	 * "The minimum amount of time that OSPM must wait after the completion
> +	 * of a command before issuing the next command, in microseconds"
> +	 */
> +	if (pcc_subspace->pcc_mrtt) {
> +		time_delta = ktime_us_delta(ktime_get(), last_cmd_cmpl_time);
> +		if (pcc_subspace->pcc_mrtt > time_delta)
> +			udelay(pcc_subspace->pcc_mrtt - time_delta);
> +	}
> +
> +	/*
> +	 * Handle the non-zero Maximum Periodic Access Rate(MPAR)
> +	 * "The maximum number of periodic requests that the subspace channel can
> +	 * support, reported in commands per minute. 0 indicates no limitation."
> +	 *
> +	 * This parameter should be ideally zero or large enough so that it can
> +	 * handle maximum number of requests that all the cores in the system can
> +	 * collectively generate. If it is not, we will follow the spec and just
> +	 * not send the request to the platform after hitting the MPAR limit in
> +	 * any 60s window
> +	 */
> +	if (pcc_subspace->pcc_mpar) {
> +		if (mpar_count == 0) {
> +			time_delta = ktime_ms_delta(ktime_get(), last_mpar_reset);
> +			if (time_delta < 60 * MSEC_PER_SEC) {
> +				dev_dbg(ras2_ctx->dev,
> +					"PCC cmd not sent due to MPAR limit");
> +				return -EIO;
> +			}
> +			last_mpar_reset = ktime_get();
> +			mpar_count = pcc_subspace->pcc_mpar;
> +		}
> +		mpar_count--;
> +	}
> +
> +	/* Write to the shared comm region. */
> +	writew_relaxed(cmd, &generic_comm_base->command);
> +
> +	/* Flip CMD COMPLETE bit */
> +	writew_relaxed(0, &generic_comm_base->status);
> +
> +	/* Ring doorbell */
> +	ret = mbox_send_message(pcc_channel, &cmd);
> +	if (ret < 0) {
> +		dev_err(ras2_ctx->dev,
> +			"Err sending PCC mbox message. cmd:%d, ret:%d\n",
> +			cmd, ret);
> +		return ret;
> +	}
> +
> +	/*
> +	 * If Minimum Request Turnaround Time is non-zero, we need
> +	 * to record the completion time of both READ and WRITE
> +	 * command for proper handling of MRTT, so we need to check
> +	 * for pcc_mrtt in addition to CMD_READ
> +	 */
> +	if (cmd == RAS2_PCC_CMD_EXEC || pcc_subspace->pcc_mrtt) {
> +		ret = ras2_check_pcc_chan(pcc_subspace);
> +		if (pcc_subspace->pcc_mrtt)
> +			last_cmd_cmpl_time = ktime_get();
> +	}
> +
> +	if (pcc_channel->mbox->txdone_irq)
> +		mbox_chan_txdone(pcc_channel, ret);
> +	else
> +		mbox_client_txdone(pcc_channel, ret);
> +
> +	return 0;


As of now, this code doesn't consider errors occurring after the
mbox_send_message.
Checking errors is, probably, necessary to take appropriate action - such as
reporting an error to the user.  Also, error or no, it is important to
call the txdone bits.

One simple solution that works is to modify the previous "return 0" to
look like:
	return ret >= 0 ? 0 : ret;

> +}
> +EXPORT_SYMBOL_GPL(ras2_send_pcc_cmd);