Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 6 Dec 2022 18:41:34 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     <alison.schofield@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Ira Weiny <ira.weiny@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ingo Molnar <mingo@redhat.com>
CC:     Alison Schofield <alison.schofield@intel.com>,
        <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v3 2/6] cxl/mbox: Add GET_POISON_LIST mailbox command
Message-ID: <638ffd5eaa3a7_c95729426@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <cover.1668115235.git.alison.schofield@intel.com>
 <46c7c7339224744fce424b196da3e5566effec17.1668115235.git.alison.schofield@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <46c7c7339224744fce424b196da3e5566effec17.1668115235.git.alison.schofield@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6wBH1uQnPW4qu9wPlcU4fp+/5pbBhOGAQlrpnvaZKsmwWpBmdlsqDjrDOnjT?=
 =?us-ascii?Q?07chTEHGyQN7GdLqe9f/yzbleMvLuBhIyue4aT1cw8ETHV332xscSmy06xea?=
 =?us-ascii?Q?oHng01BrWvDkHp257cUtP4/Aan2fRdmsn65OahUcjv2SSfI3ORm1zxQu4ccJ?=
 =?us-ascii?Q?80SsKt5Hevm/zC3NYQ3Ch/D+W85QO4JvIQEHcdCao+raG8TD/RJbs13BmBRz?=
 =?us-ascii?Q?SzR5Tk9H5a1x0t3rY2hSOP6wG1FjAhMcdeoJiqOKQ7iKoRXUiSLeY2OuhNjp?=
 =?us-ascii?Q?8erh5KV8NpNM55Z73hL2dcTmATRuhyR7DIU0D8izMK8pi66j1s4GtiahxqhK?=
 =?us-ascii?Q?V9fSF4hThNcFHzHt+lz/t8jF7WQI2C24CYMMO9Fabs1Bhj3bVGOYJK7fOpmv?=
 =?us-ascii?Q?WxvTbeLZUzD2NwTs8MeGaCXoRwUS9bQa0yJCAYDzHM0JoD7idV/o0D4+dSq6?=
 =?us-ascii?Q?RckdnSP8S6rGeNfo8+vIYSwbVAuRBAfk4Wl+v12EGNz4j72BtvfSP1G3zWyj?=
 =?us-ascii?Q?Tq1YexyphfRjKIcOFAGtxONeMVEVFh7GTkUqaszyjwldYM6UHG9iqpavu6P1?=
 =?us-ascii?Q?VBvIlFiQKQu3NckPaGN3Fd2zj6B/DGavFwZzEwEMqIwxcW4hrXhDD4xa66Ra?=
 =?us-ascii?Q?MlWupzphCu+CZNP31CXhgVS3tz0M1QEzzB9oC3spqD7lqxm/D+bKx/Wrsgjr?=
 =?us-ascii?Q?i01sRdBg9kPfFpPbbHV1CpqXle1uadqnrgpHls+a4Ac4po++WoTTtA+0YfZw?=
 =?us-ascii?Q?Fx6yfFKobw8DuZ1Em1ydvavImalmz/TXPAhn9wKfeP6/7yEHAwVgaMYCivDh?=
 =?us-ascii?Q?OHK4z7N8WJd5AU46ASdujg58KyUpoKLPCALMqELA7RpcXiTguFGCHcSLz7ND?=
 =?us-ascii?Q?d9B9gjaE6HgxLPbK5nubL+yWTOQ4dbcErQXnQDYD6Am4iBzh2MepNzEdgLLM?=
 =?us-ascii?Q?vq+sM410AIOzr1LBo3TEw8p8rHdYTasAlRH0bsCmtIfVBXIYlq/CgzoBH+Fs?=
 =?us-ascii?Q?rLbQ8K2ITWbG+rYECpTAgaaQSOhLt+BJKtZiAMkDdcUl0iSUF8lTQLCSuWRB?=
 =?us-ascii?Q?F/r+7rZxhshZZj5OPphXh0tSgnP12ud6wPIpa8K85F7Nw4ufEUMgSzFY1Dwe?=
 =?us-ascii?Q?R4k+wLuZWQxTbP9BYHYy417wyem/9yW4Pg6zRR6MpOjKn9EFx4LRaek/eeBI?=
 =?us-ascii?Q?3kF4W/+n4A+nqgUDDmUOejbTDuL3nlm67cQYk0j/UcUBS7tBX3e5G5l+PULh?=
 =?us-ascii?Q?9XCs+poxje3aYq1JFh3XXkWrkV0NQ7JR00cFsdB5i9oPwxb0jcT6p6Q4w+FZ?=
 =?us-ascii?Q?n659rE8LX4aHsw0hJm+rLjhjhl/hOGjw2mmvOwFvEOiF3CnRA3jC1PaKdemZ?=
 =?us-ascii?Q?Bg1dSagkmporRpn5Pfn27MwLGOXW4wAJUeqQZnJeJnd7TdXTXfRuaxjYtq9R?=
 =?us-ascii?Q?XKXiqlrWUg5df6ROKADofZF/J9NQLK7wCKUcKrNpua0EEtHnqiV28lk3Em3h?=
 =?us-ascii?Q?QI/UCt9V/ykQqTtSwgAOEXNgMEgnyIII1xexiC1fOYArOgwx2/3uNxprQ4Bg?=
 =?us-ascii?Q?K7pEs4c1iWlQsPUha1Kt/Lf0UwV3OIVy0+dqxHN0IkoWv755gG3l6ftSC/cY?=
 =?us-ascii?Q?Xw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 692f7966-70c6-4aa3-c0b5-08dad7fc90c6
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Dec 2022 02:41:40.0145
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LyKPbdiGnizU5F3VGxCAOX0HE3VbKJED0Luv3Xi6hzjF+8y2FjDsLtUOe8vTM7yP6cf25rdklt3/aAw1fw8kKoc6DPitu3+oxLHQa76fASM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6496
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

alison.schofield@ wrote:
> From: Alison Schofield <alison.schofield@intel.com>
> 
> CXL devices maintain a list of locations that are poisoned or result
> in poison if the addresses are accessed by the host.
> 
> Per the spec (CXL 3.0 8.2.9.8.4.1), the device returns this Poison
> list as a set of  Media Error Records that include the source of the
> error, the starting device physical address and length. The length is
> the number of adjacent DPAs in the record and is in units of 64 bytes.
> 
> Retrieve the list and log each Media Error Record as a trace event of
> type 'cxl_poison'.
> 
> When the poison list is requested by region, include the region name
> and uuid in the trace event.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> ---
>  drivers/cxl/core/mbox.c | 81 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxlmem.h    | 37 +++++++++++++++++++
>  2 files changed, 118 insertions(+)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 0c90f13870a4..88f034e97812 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -9,6 +9,9 @@
>  
>  #include "core.h"
>  
> +#define CREATE_TRACE_POINTS
> +#include <trace/events/cxl.h>
> +
>  static bool cxl_raw_allow_all;
>  
>  /**
> @@ -752,6 +755,7 @@ int cxl_dev_state_identify(struct cxl_dev_state *cxlds)
>  {
>  	/* See CXL 2.0 Table 175 Identify Memory Device Output Payload */
>  	struct cxl_mbox_identify id;
> +	__le32 val = 0;
>  	int rc;
>  
>  	rc = cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_IDENTIFY, NULL, 0, &id,
> @@ -771,6 +775,9 @@ int cxl_dev_state_identify(struct cxl_dev_state *cxlds)
>  	cxlds->lsa_size = le32_to_cpu(id.lsa_size);
>  	memcpy(cxlds->firmware_version, id.fw_revision, sizeof(id.fw_revision));
>  
> +	memcpy(&val, id.poison_list_max_mer, 3);

I see Jonathan already commented on the need for get_unaligned_le24()
here, seconded.


> +	cxlds->poison_max = min_t(u32, le32_to_cpu(val), CXL_POISON_LIST_MAX);
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_dev_state_identify, CXL);
> @@ -835,6 +842,79 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds)
>  }
>  EXPORT_SYMBOL_NS_GPL(cxl_mem_create_range_info, CXL);
>  
> +static void cxl_trace_poison(struct cxl_mbox_poison_payload_out *po,
> +			     struct cxl_region *cxlr,
> +			     const char *memdev_name,
> +			     const char *pcidev_name)

Type-safety please. Pass a 'struct cxl_memdev *' and 'struct pci_dev *'.
Might need to be 'struct device *' instead of 'struct pci_dev *'
depending on if this needs to be called from cxl_test, but anything is
better than a non-specific string.

> +{
> +	const char *region_name = cxlr ? dev_name(&cxlr->dev) : NULL;

...and push this conversion into the trace point.

> +	struct cxl_region_params *p = cxlr ? &cxlr->params : NULL;
> +	uuid_t *uuid = p ? &p->uuid : NULL;
> +	u64 addr, dpa, overflow_t = 0;
> +	u8 source;
> +	u32 len;
> +
> +	if (po->flags & CXL_POISON_FLAG_OVERFLOW)
> +		overflow_t = le64_to_cpu(po->overflow_timestamp);
> +
> +	for (int i = 0; i < le16_to_cpu(po->count); i++) {
> +		len = le32_to_cpu(po->record[i].length) * CXL_POISON_LEN_MULT;
> +		addr = le64_to_cpu(po->record[i].address);
> +		source = addr & CXL_POISON_SOURCE_MASK;
> +		dpa = addr & CXL_POISON_START_MASK;
> +
> +		trace_cxl_poison(memdev_name, pcidev_name, region_name, uuid,
> +				 dpa, len, source, po->flags, overflow_t);
> +	}
> +}
> +
> +int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> +		       struct cxl_region *cxlr)
> +{
> +	struct cxl_dev_state *cxlds = cxlmd->cxlds;
> +	const char *memdev_name = dev_name(&cxlmd->dev);
> +	const char *pcidev_name = dev_name(cxlds->dev);
> +	struct cxl_mbox_poison_payload_out *po;
> +	struct cxl_mbox_poison_payload_in pi;
> +	int nr_records = 0;
> +	int rc;
> +
> +	po = kvmalloc(cxlds->payload_size, GFP_KERNEL);
> +	if (!po)
> +		return -ENOMEM;
> +
> +	pi.offset = cpu_to_le64(offset);
> +	pi.length = cpu_to_le64(len);
> +
> +	rc = mutex_lock_interruptible(&cxlds->poison_list_mutex);

So I do not know what this mutex is protecting if there is an allocation
per cxl_mem_get_poison() invocation. Although I suspect that's somewhat
wasteful. Just allocate one buffer at the beginning of time and then use
the lock to protect that buffer.

Although, I wonder if this and Event handling should share locks and one
preallocated buffer? Otherwise I do think it is important for Events and
Poison handling to be able to make forward progress without needing to
allocate up to a megabyte of memory at runtime. The other payload_size
allocations are for one-off things that run at the beginning of time,
but Poison and Events run repeatedly.

> +	if (rc)
> +		goto out;
> +
> +	do {
> +		rc = cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_GET_POISON, &pi,
> +				       sizeof(pi), po, cxlds->payload_size);
> +		if (rc)
> +			break;
> +
> +		if (trace_cxl_poison_enabled())
> +			cxl_trace_poison(po, cxlr, memdev_name, pcidev_name);
> +
> +		/* Protect against an uncleared _FLAG_MORE */
> +		nr_records = nr_records + le16_to_cpu(po->count);
> +		if (nr_records >= cxlds->poison_max) {
> +			dev_dbg(&cxlmd->dev, "Max Error Records reached: %d\n",
> +				nr_records);
> +			break;
> +		}
> +	} while (po->flags & CXL_POISON_FLAG_MORE);
> +
> +	mutex_unlock(&cxlds->poison_list_mutex);
> +out:
> +	kvfree(po);
> +	return rc;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_mem_get_poison, CXL);
> +
>  struct cxl_dev_state *cxl_dev_state_create(struct device *dev)
>  {
>  	struct cxl_dev_state *cxlds;
> @@ -846,6 +926,7 @@ struct cxl_dev_state *cxl_dev_state_create(struct device *dev)
>  	}
>  
>  	mutex_init(&cxlds->mbox_mutex);
> +	mutex_init(&cxlds->poison_list_mutex);
>  	cxlds->dev = dev;
>  
>  	return cxlds;
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index 669868cc1553..49d891347e39 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -192,6 +192,8 @@ struct cxl_endpoint_dvsec_info {
>   *                (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
>   * @lsa_size: Size of Label Storage Area
>   *                (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
> + * @poison_max: maximum media error records held in device cache
> + * @poison_list_mutex: Mutex to synchronize poison list retrieval
>   * @mbox_mutex: Mutex to synchronize mailbox access.
>   * @firmware_version: Firmware version for the memory device.
>   * @enabled_cmds: Hardware commands found enabled in CEL.
> @@ -224,6 +226,8 @@ struct cxl_dev_state {
>  
>  	size_t payload_size;
>  	size_t lsa_size;
> +	u32 poison_max;
> +	struct mutex poison_list_mutex;  /* Protect reads of poison list */
>  	struct mutex mbox_mutex; /* Protects device mailbox and firmware */
>  	char firmware_version[0x10];
>  	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
> @@ -349,6 +353,37 @@ struct cxl_mbox_set_partition_info {
>  
>  /* Get Poison List  CXL 3.0 Spec 8.2.9.8.4.1 */
>  
> +struct cxl_mbox_poison_payload_in {
> +	__le64 offset;
> +	__le64 length;
> +} __packed;
> +
> +struct cxl_mbox_poison_payload_out {
> +	u8 flags;
> +	u8 rsvd1;
> +	__le64 overflow_timestamp;
> +	__le16 count;
> +	u8 rsvd2[0x14];

Let's use decimal values for size.

> +	struct cxl_poison_record {
> +		__le64 address;
> +		__le32 length;
> +		__le32 rsvd;
> +	} __packed record[];
> +} __packed;
> +
> +/*
> + * Get Poison List address field encodes the starting
> + * address of poison, and the source of the poison.
> + */
> +#define CXL_POISON_START_MASK		GENMASK_ULL(63, 6)
> +#define CXL_POISON_SOURCE_MASK		GENMASK(2, 0)
> +
> +/* Get Poison List record length is in units of 64 bytes */
> +#define CXL_POISON_LEN_MULT	64
> +
> +/* Kernel defined maximum for a list of poison errors */
> +#define CXL_POISON_LIST_MAX	1024
> +
>  /* Get Poison List: Payload out flags */
>  #define CXL_POISON_FLAG_MORE            BIT(0)
>  #define CXL_POISON_FLAG_OVERFLOW        BIT(1)
> @@ -395,6 +430,8 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
>  struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
>  void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
>  void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
> +int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> +		       struct cxl_region *cxlr);
>  #ifdef CONFIG_CXL_SUSPEND
>  void cxl_mem_active_inc(void);
>  void cxl_mem_active_dec(void);
> -- 
> 2.37.3
>