Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 13 Dec 2022 10:56:43 -0800
From:   Ira Weiny <ira.weiny@intel.com>
To:     johnny <johnny.li@montage-tech.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Alison Schofield <alison.schofield@intel.com>,
        "Vishal Verma" <vishal.l.verma@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dave Jiang <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-pci@vger.kernel.org>,
        <linux-acpi@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH V4 2/9] cxl/mem: Read, trace, and clear events on driver
 load
Message-ID: <Y5jK658xKJC3bLny@iweiny-mobl>
References: <20221212070627.1372402-1-ira.weiny@intel.com>
 <20221212070627.1372402-3-ira.weiny@intel.com>
 <Y5ggXi88HancP2LZ@SR651>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y5ggXi88HancP2LZ@SR651>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?TP7kCbyxrkSccXqGttWafJAUfOfiOz6njoDR4XSNpgINJvQEXOpEEScItNzD?=
 =?us-ascii?Q?r2tpjBeBkWjUUjY0gL4hpDI2f43h8ae4FTZo1DsuR3TQV1D1koMvNqlR6XWf?=
 =?us-ascii?Q?6TayAZRnJzQXXMnCP7XXmdw91BHA3LhGp9RkagUD05GR0WJ87vpZ7+5vxx69?=
 =?us-ascii?Q?LuWgKxp4HXJWKqGTGne/NGeJtOMetAnspxRxpAgtIRcmLPAl2yj+F9G7oap+?=
 =?us-ascii?Q?SIA9cHk24V46kSLkG67vb2uLhwS8HTuGjLnpMaclV8+ZjUjWpa745kt0/GHA?=
 =?us-ascii?Q?QeYOsoziU6YzTWAUGQbn1a+uO+D8gCxTAdSIkTCK9u5PWqMpwCb+sWeBEmz0?=
 =?us-ascii?Q?VB85r+5q8iIEv4CJMf7dbCYxp+OdEVK9Je08na029vVTcBOHhzRHakRGRqLr?=
 =?us-ascii?Q?sa/HXfLyNjXhqRErI/+Ag/jh493Z5p3i4lsPlrXLSJf8EGUKCE88wQpn+8ay?=
 =?us-ascii?Q?z8wd1xFFpiWf21m5lXPx2YAGVXJlWP3y8bJUbQ/4cNwl/kJkwzPDQ/TXJK+C?=
 =?us-ascii?Q?6jTJX1sXcDU/XLSEWs+S2YpqpBy1E2Ap3o2SqYCKzyp06uO1slMZMjzZR3Y/?=
 =?us-ascii?Q?9p4ooZkxOIfdabGqLZtKliql/vUtYSiToqSU2u0A0cc5cVRhndzpPyIpE+DB?=
 =?us-ascii?Q?q2Ji+l9+rLzoQmK9oCPzaxg6mIdpwIvsxpK8IQxxcsPibKC4LFkr3N8enUpL?=
 =?us-ascii?Q?tVoO7GZgAu1mKPd5TaQUzddhL3rgc5kJJwoNx7KQvw0AS0/ee5Ldn3h0yMal?=
 =?us-ascii?Q?+fSD5tBo6ic4Q6Qk3oGBmTi2QVgOHMEde4dym+0HArHOpsFIPxYtbCndzO/4?=
 =?us-ascii?Q?JMbiaMtUJ2N0rLx47/5Bvu/VCFvwmx+QMlCaHEVsB3Ev/mgfuRuQzXm0rves?=
 =?us-ascii?Q?A3Zvesndmkt8F3g2Kvr/gZEPb41Vb7lzDsEnuI8NcKXkC3YKYEHIvzqsuTYf?=
 =?us-ascii?Q?IqZUZxpgsfdlT5+GQ6WxNt9lTRHyu/Qr6/Eg560DNVz5ZwawV/9K2EC3dxoo?=
 =?us-ascii?Q?3fuTUDsZT3QggOO8jAiI82IT+rDscrniLjnQZOlTdq8sLotxaF5NvtUE0pSt?=
 =?us-ascii?Q?Et7jdfewvNBtlRV/5P/l6ader7QE0Qw1ifctUs6AbZK9MiKZmAFyRcoZUNnf?=
 =?us-ascii?Q?+7s5OXG6GTmGPvNbRtuzKHNsccM063oTeAzifxkaHgeh5IQub/fGY00iPq30?=
 =?us-ascii?Q?eNF2ggaloS0cOyTfaqKGf0m08XVWF6IwWLtwedJh0N/G/LqV/u3T0Axkqnje?=
 =?us-ascii?Q?Wi0l5UGWTnuXcOygrjsayz//3GvMz9IL1KoYpAN11Ijj0k2UOfycgBAQupAO?=
 =?us-ascii?Q?j+2kVVDG+262fCpBzW2TruYyynPl0QzIPPgPmGUdUEWBk29nOwIkJEtPHS6+?=
 =?us-ascii?Q?t3I1JsWbtWySPUpgq0Eng/vnNEcjlBzAjRq3e3KzFqmoXGOWKNEFolX2DUjD?=
 =?us-ascii?Q?b9pqWAC12CHOh7pY6dI6sAY4qtgHVO5e00Xb4vYu8899aQskMYTwFaTJRVZ5?=
 =?us-ascii?Q?aUETjuZRCGmtD3iqeiBFPfuz6kdM4K4bu9n+4yd7jWEZ83xLktITHJpFu9EI?=
 =?us-ascii?Q?imyKbgkQstpY2KXHaQlqrRCM0NIplJB2Bpet4A1U?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 43d7db74-a917-4b25-b587-08dadd3bc994
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2022 18:56:48.2840
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: skgCOQ9Kpl8xe2aEnd9IZEUuXRRzBk3z0a4k38CyRVALlG2+gajfnyV6T6WIiNXRrE5ITo68CxtXXJ9ajr0F2w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6567
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Dec 13, 2022 at 02:49:02PM +0800, johnny wrote:
> On Sun, Dec 11, 2022 at 11:06:20PM -0800, ira.weiny@intel.com wrote:
> > From: Ira Weiny <ira.weiny@intel.com>
> > 

[snip]

> > +
> > +#define CXL_EVENT_RECORD_DATA_LENGTH 0x50
> > +struct cxl_event_record_raw {
> > +	struct cxl_event_record_hdr hdr;
> > +	u8 data[CXL_EVENT_RECORD_DATA_LENGTH];
> > +} __packed;
> > +
> > +/*
> > + * Get Event Records output payload
> > + * CXL rev 3.0 section 8.2.9.2.2; Table 8-50
> > + */
> > +#define CXL_GET_EVENT_FLAG_OVERFLOW		BIT(0)
> > +#define CXL_GET_EVENT_FLAG_MORE_RECORDS		BIT(1)
> I don't see any code consumes this more flag, is anything I miss?
> Device shall set this more flag when single output payload can not fit in all records

I should have removed this flag and put something in the cover letter.  I left
it in for completeness but you are correct it is unused.

We determined back in V1 that the more bit was useless in this particular
looping of Get Events Records.[1]

The net-net is that if the driver does not see the number of records go to 0 it
can't be sure it will get an interrupt for the next set of events.  Therefore
it loops until it sees the number of records go to 0.

Ira

[1] https://lore.kernel.org/all/Y4blpk%2FesXJMe79Y@iweiny-desk3/

> > +struct cxl_get_event_payload {
> > +	u8 flags;
> > +	u8 reserved1;
> > +	__le16 overflow_err_count;
> > +	__le64 first_overflow_timestamp;
> > +	__le64 last_overflow_timestamp;
> > +	__le16 record_count;
> > +	u8 reserved2[10];
> > +	struct cxl_event_record_raw records[];
> > +} __packed;
> > +
> > +/*
> > + * CXL rev 3.0 section 8.2.9.2.2; Table 8-49
> > + */
> > +enum cxl_event_log_type {
> > +	CXL_EVENT_TYPE_INFO = 0x00,
> > +	CXL_EVENT_TYPE_WARN,
> > +	CXL_EVENT_TYPE_FAIL,
> > +	CXL_EVENT_TYPE_FATAL,
> > +	CXL_EVENT_TYPE_MAX
> > +};
> > +
> > +/*
> > + * Clear Event Records input payload
> > + * CXL rev 3.0 section 8.2.9.2.3; Table 8-51
> > + */
> > +#define CXL_CLEAR_EVENT_MAX_HANDLES (0xff)
> > +struct cxl_mbox_clear_event_payload {
> > +	u8 event_log;		/* enum cxl_event_log_type */
> > +	u8 clear_flags;
> > +	u8 nr_recs;
> > +	u8 reserved[3];
> > +	__le16 handle[CXL_CLEAR_EVENT_MAX_HANDLES];
> > +} __packed;
> > +#define CXL_CLEAR_EVENT_LIMIT_HANDLES(payload_size)			\
> > +	(((payload_size) -						\
> > +		(sizeof(struct cxl_mbox_clear_event_payload) -		\
> > +		 (sizeof(__le16) * CXL_CLEAR_EVENT_MAX_HANDLES))) /	\
> > +		sizeof(__le16))
> > +
> >  struct cxl_mbox_get_partition_info {
> >  	__le64 active_volatile_cap;
> >  	__le64 active_persistent_cap;
> > @@ -441,6 +524,7 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
> >  struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
> >  void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
> >  void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
> > +void cxl_mem_get_event_records(struct cxl_dev_state *cxlds, u32 status);
> >  #ifdef CONFIG_CXL_SUSPEND
> >  void cxl_mem_active_inc(void);
> >  void cxl_mem_active_dec(void);
> > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > index 3a66aadb4df0..a2d8382bc593 100644
> > --- a/drivers/cxl/pci.c
> > +++ b/drivers/cxl/pci.c
> > @@ -417,8 +417,37 @@ static void disable_aer(void *pdev)
> >  	pci_disable_pcie_error_reporting(pdev);
> >  }
> >  
> > +static void cxl_mem_free_event_buffer(void *buf)
> > +{
> > +	kvfree(buf);
> > +}
> > +
> > +/*
> > + * There is a single buffer for reading event logs from the mailbox.  All logs
> > + * share this buffer protected by the cxlds->event_log_lock.
> > + */
> > +static int cxl_mem_alloc_event_buf(struct cxl_dev_state *cxlds)
> > +{
> > +	struct cxl_get_event_payload *buf;
> > +
> > +	dev_dbg(cxlds->dev, "Allocating event buffer size %zu\n",
> > +		cxlds->payload_size);
> > +
> > +	buf = kvmalloc(cxlds->payload_size, GFP_KERNEL);
> > +	if (!buf)
> > +		return -ENOMEM;
> > +
> > +	if (devm_add_action_or_reset(cxlds->dev, cxl_mem_free_event_buffer,
> > +				     buf))
> > +		return -ENOMEM;
> > +
> > +	cxlds->event.buf = buf;
> > +	return 0;
> > +}
> > +
> >  static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> >  {
> > +	struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus);
> >  	struct cxl_register_map map;
> >  	struct cxl_memdev *cxlmd;
> >  	struct cxl_dev_state *cxlds;
> > @@ -494,6 +523,17 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> >  	if (IS_ERR(cxlmd))
> >  		return PTR_ERR(cxlmd);
> >  
> > +	rc = cxl_mem_alloc_event_buf(cxlds);
> > +	if (rc)
> > +		return rc;
> > +
> > +	/*
> > +	 * When BIOS maintains CXL error reporting control, it will process
> > +	 * event records.  Only one agent can do so.
> > +	 */
> > +	if (host_bridge->native_cxl_error)
> > +		cxl_mem_get_event_records(cxlds, CXLDEV_EVENT_STATUS_ALL);
> > +
> >  	if (cxlds->regs.ras) {
> >  		pci_enable_pcie_error_reporting(pdev);
> >  		rc = devm_add_action_or_reset(&pdev->dev, disable_aer, pdev);
> > -- 
> > 2.37.2
> > 
> > 
>