Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 9 Dec 2022 14:33:20 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Ira Weiny <ira.weiny@intel.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     Bjorn Helgaas <bhelgaas@google.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dave Jiang <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-pci@vger.kernel.org>,
        <linux-acpi@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH V3 1/8] cxl/mem: Read, trace, and clear events on driver
 load
Message-ID: <6393b7b0e4953_579c1294af@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221208052115.800170-1-ira.weiny@intel.com>
 <20221208052115.800170-2-ira.weiny@intel.com>
 <639376e1452bd_579c12945c@dwillia2-xfh.jf.intel.com.notmuch>
 <Y5Oh89JFAWYIITUV@iweiny-desk3>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y5Oh89JFAWYIITUV@iweiny-desk3>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Amyq1iQuccmUpUoxnftiOP+I2yMQ+bdC4BP9wfIs2atl81qCKqVwY5bIR/9W?=
 =?us-ascii?Q?NdhxOKYNpWa9Zw43fpg2dxh+GvIyrltLkwk0mpiC7mnT7ZAaf4efB6J0eZeg?=
 =?us-ascii?Q?Qq59IPdxqzqpm0OyFjMbpW7C9e2fZw7XNE1D4YDL9HqPUttEDkmhiS1M9ebl?=
 =?us-ascii?Q?vxf24RrAB2estgEIPlpN44W55SFlWZxFAgh0+BwkL+oJg6fAdBQKGLp7WZFL?=
 =?us-ascii?Q?j/hrKa1k9eIIHur8+Wg/NvwHSYuchmwRmZM7xHLr0pv3Z9kisa3diX6W1yBf?=
 =?us-ascii?Q?Ci3K/BeTXIb/RpmKJcopu4Ld1RRSbno+4i3Rjd+LNFDk8gu+kqGrG11QQ7Hv?=
 =?us-ascii?Q?hCXX/wUF6W62EPK7MsU7/gL7T4hHjN18f88CC/r7xRjj2+na3sJ8YGA1URDV?=
 =?us-ascii?Q?ox8sVYzpMgjAARmo/LLBr0Nrm2LHEJl1o4DLpMqYnFWy06zEO+qQVF8BZKrY?=
 =?us-ascii?Q?7Ej+MqNejvtdCrqqERma2tOCQ3wdoYoWcr0ZqT+N3m8fatxJO+P+dPVmZlfo?=
 =?us-ascii?Q?UiDh1yvlqwREjkRcdjF+CcuWaaW6xpCNgMWfsr1jSSvwy7bJhuYQK+onUlru?=
 =?us-ascii?Q?MQnQQVl5cUIrPs9SIyRA4Kg/mFPnCvWqxyVbCYjzTiq1ByqlnCtcnMkSefBt?=
 =?us-ascii?Q?wYP2O2EAuftGxjwNHYSL8mCrEVkzJ4lefGZsd+ECuz1Iei9t8ew2fJKyswN7?=
 =?us-ascii?Q?KAOieoHIm06lY8hx5+SqsxiWpc2IjLxHRAULh/RU8BJabvauSDaiidcbuvjx?=
 =?us-ascii?Q?ZmTUW4X+ywTRMzdvJHQ+A7g27HlLr3k0aakz23v13a+feKPpzpnU2pcSVU3g?=
 =?us-ascii?Q?W8Vye2Sxp3qRId3vRflen1jGyrG7AAC7COtgCGq9BdiH2DayhjfWTvc0YplO?=
 =?us-ascii?Q?qrk7qYScOPWd8GgvvWv5TKThzeP4gx6CgZS71Hur9cfyWagwbRyzz+DbY0uY?=
 =?us-ascii?Q?rPjX7BHzC5xaS3knzTVJ/UcazMQOtboQQ5HcCQ9pbGttbfkhHXx0vrzGpqY6?=
 =?us-ascii?Q?HRgj3iBH9IvTWE/Jz7OzbN0qMMIv8izNT3ahb4Tp6AGSx1OXoJSbIsY5/bdC?=
 =?us-ascii?Q?5OJWS64m6VZ4Xw27JEsaQPIYMNAhG0ZZH7Yv6UfLUklxuu3QkDSBAfIvdXp6?=
 =?us-ascii?Q?ppsB1d/EIQCdvmEvGEku6jkf0arI5NY7qd7FvRA/h6Uc6iH1g4TongNi429v?=
 =?us-ascii?Q?GuiHrFHwlKP+wYuhsbdKitVCskzCCTGuV6PiIZYI03C2H3x3q3G3mHLcCJFg?=
 =?us-ascii?Q?+iiHYL005BCbbBkoC6nWupbPEu1rsIt/ibeWjZkDshgzvzmeqoIQY4nJLQg8?=
 =?us-ascii?Q?1oWhY2mzTPm5HMCBZo+APp+UQdWLJ3VDn0/vClCbSolD8oJpMcIyby2giAAH?=
 =?us-ascii?Q?c4WqKC3MQ9qE84mEN8Yld5KSBBimhlirLbnD+zVHutxg7BP3Dx0iNhJ+BcDO?=
 =?us-ascii?Q?QhDXI7AMb55t7U2XiSY4aP/NmSxAeTabxHR5Q2b/UeB7z7DWXw27jNJ97Aon?=
 =?us-ascii?Q?QLAR5MQaXmWQRdOv39oIBS8/HFw1TFMdxNsrzfoXD1E0zIPBJspwbmeN5ntT?=
 =?us-ascii?Q?eTmjWyMARtQ6qv9tYVRCw1vVQzBp45zaN6b/o1/UFGHw2GZHBcRJNgd8nrdz?=
 =?us-ascii?Q?4Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 720091e0-6160-447f-72cd-08dada3561b9
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Dec 2022 22:33:23.6355
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XD/SYv6b0O1fRjZP1ArEeD2V1NnUmDoSptl2VNrgLO9esdA2YVrMiUXiOJNQm7yNmG+tAJ7K0IhatVdX7cb0MfoU6rte97e3vDnNiorJrcc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5355
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Ira Weiny wrote:
[..]
> > > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > > index 3a66aadb4df0..86c84611a168 100644
> > > --- a/drivers/cxl/pci.c
> > > +++ b/drivers/cxl/pci.c
> > > @@ -417,8 +417,44 @@ static void disable_aer(void *pdev)
> > >  	pci_disable_pcie_error_reporting(pdev);
> > >  }
> > >  
> > > +static void cxl_mem_free_event_buffer(void *buf)
> > > +{
> > > +	kvfree(buf);
> > > +}
> > > +
> > > +/*
> > > + * There is a single buffer for reading event logs from the mailbox.  All logs
> > > + * share this buffer protected by the cxlds->event_log_lock.
> > > + */
> > > +static void cxl_mem_alloc_event_buf(struct cxl_dev_state *cxlds)
> > > +{
> > > +	struct cxl_get_event_payload *buf;
> > > +
> > > +	dev_dbg(cxlds->dev, "Allocating event buffer size %zu\n",
> > > +		cxlds->payload_size);
> > > +
> > > +	buf = kvmalloc(cxlds->payload_size, GFP_KERNEL);
> > > +	if (WARN_ON_ONCE(!buf))
> > 
> > No, why is event init so special that it behaves differently than all
> > the other init-time allocations this driver does?
> 
> Previous review agreed that a warn on once would be printed if this universal
> buffer was not allocated.
> 
> > 
> > > +		return;
> > 
> > return -ENOMEM;
> > 
> > > +
> > > +	if (WARN_ON_ONCE(devm_add_action_or_reset(cxlds->dev,
> > > +			 cxl_mem_free_event_buffer, buf)))
> > > +		return;
> > 
> > ditto.
> 
> I'll change both of these with a dev_err() and bail during init.

No real need to dev_err() for a simple memory allocation faliure, but
at least it is better than a WARN

> 
> > 
> > > +
> > > +	cxlds->event.buf = buf;
> > > +}
> > > +
> > > +static void cxl_clear_event_logs(struct cxl_dev_state *cxlds)
> > > +{
> > > +	/* Force read and clear of all logs */
> > > +	cxl_mem_get_event_records(cxlds, CXLDEV_EVENT_STATUS_ALL);
> > > +	/* Ensure prior partial reads are handled, by starting over again */
> > 
> > What partial reads? cxl_mem_get_event_records() reads every log until
> > each returns an empty result. Any remaining events after this returns
> > are events that fired during the retrieval.
> 
> Jonathan was concerned that something could read part of the log and because of
> the statefullness of the log processing this reading of the log could start in
> the beginning.  Perhaps from a previous driver unload while reading?

The driver will not unload without completing any current executions of
the event retrieval thread otherwise that's an irq shutdown bug.

> I guess I was also thinking the BIOS could leave things this way?  But I think
> we should not be here if the BIOS was ever involved right?

If the OS has CXL Error control and all Event irqs are steered to the OS
then the driver must be allowed to assume that it has exclusive control
over event retrieval and clearing.

> > So I do not think cxl_clear_event_logs() needs to exist, just call
> > cxl_mem_get_event_records(CXLDEV_EVENT_STATUS_ALL) once and that's it.
> 
> That was my inclination but Jonathan's comments got me thinking I was wrong.

Perhaps that was before we realized the recent CXL _OSC entanglement.