Date:   Thu, 26 Jan 2023 20:39:26 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     <alison.schofield@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Dave Jiang" <dave.jiang@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Steven Rostedt" <rostedt@goodmis.org>
CC:     Alison Schofield <alison.schofield@intel.com>,
        <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>
Subject: RE: [PATCH v5 1/5] cxl/mbox: Add GET_POISON_LIST mailbox command
Message-ID: <63d3557e6b656_ea2222943f@dwillia2-xfh.jf.intel.com.notmuch>
References: <cover.1674070170.git.alison.schofield@intel.com>
 <de11785ff05844299b40b100f8e0f56c7eef7f08.1674070170.git.alison.schofield@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <de11785ff05844299b40b100f8e0f56c7eef7f08.1674070170.git.alison.schofield@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+7gThcYYBy5yKn/smkwpYTw66ec7WF494+HQg8b3fVPH77V0kgQLWhcR9eXb?=
 =?us-ascii?Q?QEf+kOqMeT3JEIjY+p1KIGZ3B0NwbZRM4PDLoppbVSkKjQ5ynY0IvPeXTRa3?=
 =?us-ascii?Q?ZXzxgIL4jMClVA8OzvOS/ylozkqEwE8EiS22qynIpMBnKEzG1uSj8vjEIlB7?=
 =?us-ascii?Q?Xi1fQ+qH5cayw+A3CpY0XQp1KUfBP8P8Xl6FyZMQ4p0zONO5vSS3R6tpwgso?=
 =?us-ascii?Q?8zLl79tzrWxE0mZ7rjzqUxLZG7xFIvRFlxhKx2GiONBNR589vxKxgQxIsm7j?=
 =?us-ascii?Q?4GCFk5nOQTckPrPL6Nu4n7gWMOMfv7zxdgkYhYRu6btNpOW84xLuqKO4dRYk?=
 =?us-ascii?Q?iQOsTAJVdrF0LqBhByCHLuzbk0EbxlrAhLIDjw87zFbl8S15Tsn0Lm/733QH?=
 =?us-ascii?Q?j6GrqdnISPs0gLMq0UiDZG/TVoV2Ah2OTSfZNIvZYJCtRVk8XuWee5ngfETb?=
 =?us-ascii?Q?FcdKbq3sqTepbDORRKsGUDE60hRf7G5Zb7K1lMA0wVMI9S1LjPh0bxN4YFOd?=
 =?us-ascii?Q?duvBBWdo9xNDI/KC9IYLOGBbfIog4WdGiFap+Ak0cBlzxw46I/HTSxj+GXsE?=
 =?us-ascii?Q?VybkfrxeM80IFWZb+QoKh+7DSwW95evpbX4K2oCXp35WO+zxw0evssfxH1Lc?=
 =?us-ascii?Q?GKU8/4ZZVO69XGzPmgERdgilESqOqD+yhq94o7XxgVKufy8Cqgl8bD6tAWz+?=
 =?us-ascii?Q?7qWuC/+Q4civIPgowH6Om+B1Q1i6qwD5avnoFLsqJM10DBYt86nu3adRJ1Km?=
 =?us-ascii?Q?+Q/A6C3bu4xit2ircItgx2vCUJUpr0PNh+x7spzlsAOoOHGTtmce162j1Eww?=
 =?us-ascii?Q?oTDwpxfqDOqo9TZn4qWbjx2dhA4/+Jhf29dHz0hK+nqAvPtfdDVVDAqQqm0R?=
 =?us-ascii?Q?POvyiMGWPOuaZK9Kv3DL/+heTZT7evRrcXfDaL9lRPhl3eSD/1hT+8VvPfcs?=
 =?us-ascii?Q?qqOni/cRF0yopfXgDJtMPkut34klLPLbFMG0VMyFDRUXcYv46Ws1ozi8SEyy?=
 =?us-ascii?Q?saza5ngkTTlSnYSMTLQqw+N0w+UBH3/ZcYqpWt7VZOy5+/wnoMnZdgKeD8zH?=
 =?us-ascii?Q?q31JWm6y+1tAvhwSPbSqZNw3y1z129ZLbxAppXETabgF99UfyVVE8rIxGfR8?=
 =?us-ascii?Q?yIAJNAc4wEE3Fh2mZxkhvWOwCaIOeoxJWFe6fhtWt1HGCUKfqvnlp5u/TkYx?=
 =?us-ascii?Q?5FQfiD2/dRRIuCUxFtNx3OK1Xq/nSHH0Y2pbbYJrGsdVZ9DkEuW0SM7tp8FA?=
 =?us-ascii?Q?Du2r3ZllOHfkRPIJMpwGqiDYrSalfHRKPXVE8556a8oemimrGvvX8zfXOo9K?=
 =?us-ascii?Q?tQ5bjpA9B+5rcOcJdjWnhHIycH7SLKgTsRirpn60dEZSxHMixCFdsKmFQ/vh?=
 =?us-ascii?Q?DA1ijtNNXD6Ew8TFJ7qobiunKHC+SDQWgd042Kt7SFnR0evZ627HA7k0J0vO?=
 =?us-ascii?Q?3L4Hj6USX9cP2yx9vFE2aoKwZn1F3BEaoCm1CZ7EhVHIStPSqcKtSRFnM2tN?=
 =?us-ascii?Q?4fxAM9PTC36OqUcqvz/YWFoVdjudKkh8IjW0rWSoaT2pBvY7ELOhc5V2mNOR?=
 =?us-ascii?Q?35VaoLz/viyW4P7rOMEipC9BPuM9W3DDENkqMhMgZuGcaIgCoAYEoyzu8S1U?=
 =?us-ascii?Q?Xw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 23120e40-f758-4f81-ac9d-08db00207a0f
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2023 04:39:29.2358
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pOD7V6NfaX3Uodf9+05Rs8oGWXCuz7PIeAg7TOU/4uErrd/ZP+TlyMTC8/lTL7beVw/uuEL/OgiCdaCh+hAoJrHRr/E6Bycj4eko7xB+yOw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6367
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

alison.schofield@ wrote:
> From: Alison Schofield <alison.schofield@intel.com>
> 
> CXL devices maintain a list of locations that are poisoned or result
> in poison if the addresses are accessed by the host.
> 
> Per the spec (CXL 3.0 8.2.9.8.4.1), the device returns this Poison
> list as a set of  Media Error Records that include the source of the
> error, the starting device physical address and length. The length is
> the number of adjacent DPAs in the record and is in units of 64 bytes.
> 
> Retrieve the poison list.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 74 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    | 67 +++++++++++++++++++++++++++++++++++++
>  drivers/cxl/pci.c       |  4 +++
>  3 files changed, 145 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index b03fba212799..a38be141e108 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -4,7 +4,9 @@
>  #include <linux/security.h>
>  #include <linux/debugfs.h>
>  #include <linux/mutex.h>
> +#include <asm/unaligned.h>
>  #include <cxlmem.h>
> +#include <cxlpci.h>
>  #include <cxl.h>
>  
>  #include "core.h"
> @@ -770,6 +772,7 @@ int cxl_dev_state_identify(struct cxl_dev_state *cxlds)
>  	/* See CXL 2.0 Table 175 Identify Memory Device Output Payload */
>  	struct cxl_mbox_identify id;
>  	struct cxl_mbox_cmd mbox_cmd;
> +	u32 val;
>  	int rc;
>  
>  	mbox_cmd = (struct cxl_mbox_cmd) {
> @@ -793,6 +796,11 @@ int cxl_dev_state_identify(struct cxl_dev_state *cxlds)
>  	cxlds->lsa_size = le32_to_cpu(id.lsa_size);
>  	memcpy(cxlds->firmware_version, id.fw_revision, sizeof(id.fw_revision));
>  
> +	if (test_bit(CXL_MEM_COMMAND_ID_GET_POISON, cxlds->enabled_cmds)) {
> +		val = get_unaligned_le24(id.poison_list_max_mer);
> +		cxlds->poison.max_mer = min_t(u32, val, CXL_POISON_LIST_MAX);

There are not many abbreviations in the subsystem, most names are
spelled out. So, s/cxlds->poison.max_mer/cxlds->poison.max_errors/, or
something like that?

> +	}
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_dev_state_identify, CXL);
> @@ -857,6 +865,72 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds)
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_mem_create_range_info, CXL);
>  
> +int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> +		       struct cxl_region *cxlr)
> +{
> +	struct cxl_dev_state *cxlds = cxlmd->cxlds;
> +	struct cxl_poison_state *cxlps = &cxlds->poison;

I think similar to the cxl_nvd / cxl_nvb concern that Dave raised,
@cxlds is too close to @cxlps. Just type out cxlds->poison.lock. No need
to call it getpoison_lock when it is already in 'struct
cxl_poison_state'.


> +	struct cxl_mbox_poison_payload_out *po;
> +	struct cxl_mbox_poison_payload_in pi;
> +	struct cxl_mbox_cmd mbox_cmd;
> +	int nr_records = 0;
> +	int rc;
> +
> +	rc = mutex_lock_interruptible(&cxlps->getpoison_lock);
> +	if (rc)
> +		return rc;
> +
> +	po = cxlps->getpoison_po;
> +	pi.offset = cpu_to_le64(offset);
> +	pi.length = cpu_to_le64(len / CXL_POISON_LEN_MULT);
> +
> +	mbox_cmd = (struct cxl_mbox_cmd) {
> +		.opcode = CXL_MBOX_OP_GET_POISON,
> +		.size_in = sizeof(pi),
> +		.payload_in = &pi,
> +		.size_out = cxlds->payload_size,
> +		.payload_out = po,
> +		.min_out = struct_size(po, record, 0),
> +	};
> +
> +	do {
> +		rc = cxl_internal_send_cmd(cxlds, &mbox_cmd);
> +		if (rc)
> +			break;
> +
> +		/* TODO TRACE the media error records */
> +
> +		/* Protect against an uncleared _FLAG_MORE */
> +		nr_records = nr_records + le16_to_cpu(po->count);
> +		if (nr_records >= cxlps->max_mer) {
> +			dev_dbg(&cxlmd->dev, "Max Error Records reached: %d\n",
> +				nr_records);
> +			break;
> +		}
> +	} while (po->flags & CXL_POISON_FLAG_MORE);
> +
> +	mutex_unlock(&cxlps->getpoison_lock);
> +	return rc;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_mem_get_poison, CXL);
> +
> +int cxl_poison_state_init(struct cxl_dev_state *cxlds)
> +{
> +	struct cxl_poison_state *cxlps = &cxlds->poison;
> +
> +	if (!test_bit(CXL_MEM_COMMAND_ID_GET_POISON, cxlds->enabled_cmds))
> +		return 0;
> +
> +	cxlps->getpoison_po = devm_kzalloc(cxlds->dev, cxlds->payload_size,
> +					   GFP_KERNEL);
> +	if (!cxlps->getpoison_po)
> +		return -ENOMEM;
> +
> +	mutex_init(&cxlps->getpoison_lock);
> +	return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_poison_state_init, CXL);
> +
>  struct cxl_dev_state *cxl_dev_state_create(struct device *dev)
>  {
>  	struct cxl_dev_state *cxlds;
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index ab138004f644..f53fae20f502 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -193,6 +193,24 @@ struct cxl_endpoint_dvsec_info {
>  	struct range dvsec_range[2];
>  };
>  
> +/**
> + * struct cxl_poison_state - Poison list retrieval
> + *
> + * @max_mer: Maximum media error records held in device cache
> + * @getpoison_po: The poison list payload returned by device
> + * @getpoison_lock: Protect reads of the poison list
> + *
> + * Reads of the poison list are synchronized to ensure that a reader
> + * does not get an incomplete list because their request overlapped
> + * (was interrupted or preceded by) another read request of the same
> + * DPA range. CXL Spec 3.0 Section 8.2.9.8.4.1
> + */
> +struct cxl_poison_state {
> +	u32 max_mer;

Suggested rename to 'max_errors' mentioned earlier.

> +	void *getpoison_po;

No need for this to be named getpoison as its already prefixed by
"cxlds->poison". This can also be a specific typed data-structure
pointer, right?

> +	struct mutex getpoison_lock;  /* Protect reads of poison list */

Suggested rename 'lock' mentioned earlier.

> +};
> +
>  /**
>   * struct cxl_dev_state - The driver device state
>   *
> @@ -211,6 +229,7 @@ struct cxl_endpoint_dvsec_info {
>   *                (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
>   * @mbox_mutex: Mutex to synchronize mailbox access.
>   * @firmware_version: Firmware version for the memory device.
> + * @poison: Poison list retrieval
>   * @enabled_cmds: Hardware commands found enabled in CEL.
>   * @exclusive_cmds: Commands that are kernel-internal only
>   * @dpa_res: Overall DPA resource tree for the device
> @@ -245,6 +264,7 @@ struct cxl_dev_state {
>  	size_t lsa_size;
>  	struct mutex mbox_mutex; /* Protects device mailbox and firmware */
>  	char firmware_version[0x10];
> +	struct cxl_poison_state poison;
>  	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
>  	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
>  
> @@ -372,6 +392,50 @@ struct cxl_mbox_set_partition_info {
>  
>  #define  CXL_SET_PARTITION_IMMEDIATE_FLAG	BIT(0)
>  
> +/* Get Poison List  CXL 3.0 Spec 8.2.9.8.4.1 */
> +struct cxl_mbox_poison_payload_in {
> +	__le64 offset;
> +	__le64 length;
> +} __packed;
> +
> +struct cxl_mbox_poison_payload_out {
> +	u8 flags;
> +	u8 rsvd1;
> +	__le64 overflow_t;
> +	__le16 count;
> +	u8 rsvd2[20];
> +	struct cxl_poison_record {
> +		__le64 address;
> +		__le32 length;
> +		__le32 rsvd;
> +	} __packed record[];
> +} __packed;
> +
> +/*
> + * Get Poison List address field encodes the starting
> + * address of poison, and the source of the poison.
> + */
> +#define CXL_POISON_START_MASK		GENMASK_ULL(63, 6)
> +#define CXL_POISON_SOURCE_MASK		GENMASK(2, 0)
> +
> +/* Get Poison List record length is in units of 64 bytes */
> +#define CXL_POISON_LEN_MULT	64
> +
> +/* Kernel defined maximum for a list of poison errors */
> +#define CXL_POISON_LIST_MAX	1024
> +
> +/* Get Poison List: Payload out flags */
> +#define CXL_POISON_FLAG_MORE            BIT(0)
> +#define CXL_POISON_FLAG_OVERFLOW        BIT(1)
> +#define CXL_POISON_FLAG_SCANNING        BIT(2)
> +
> +/* Get Poison List: Poison Source */
> +#define CXL_POISON_SOURCE_UNKNOWN	0
> +#define CXL_POISON_SOURCE_EXTERNAL	1
> +#define CXL_POISON_SOURCE_INTERNAL	2
> +#define CXL_POISON_SOURCE_INJECTED	3
> +#define CXL_POISON_SOURCE_VENDOR	7
> +
>  /**
>   * struct cxl_mem_command - Driver representation of a memory device command
>   * @info: Command information as it exists for the UAPI
> @@ -441,6 +505,9 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
>  struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
>  void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
>  void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
> +int cxl_poison_state_init(struct cxl_dev_state *cxlds);
> +int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> +		       struct cxl_region *cxlr);
>  #ifdef CONFIG_CXL_SUSPEND
>  void cxl_mem_active_inc(void);
>  void cxl_mem_active_dec(void);
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 3a66aadb4df0..d21357f3eca0 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -482,6 +482,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  	if (rc)
>  		return rc;
>  
> +	rc = cxl_poison_state_init(cxlds);
> +	if (rc)
> +		return rc;
> +
>  	rc = cxl_dev_state_identify(cxlds);
>  	if (rc)
>  		return rc;
> -- 
> 2.37.3
>