Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 29 Jun 2022 20:35:34 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Lukas Wunner <lukas@wunner.de>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Dave Jiang" <dave.jiang@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        <linux-pci@vger.kernel.org>
Subject: Re: [PATCH V12 6/9] cxl/port: Read CDAT table
Message-ID: <Yr0aBnTJtFXrCUSM@iweiny-desk3>
References: <20220628041527.742333-1-ira.weiny@intel.com>
 <20220628041527.742333-7-ira.weiny@intel.com>
 <20220628154727.0000021c@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220628154727.0000021c@Huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?5fNnQV7GRarl+k/JapNe8gY98mN0udzNgAho1Wr+wfxHDg4jBFIHPXpXMG+9?=
 =?us-ascii?Q?Ugx2IWPuDigVjT+/ENQop3x8GUm0ouhFgJAVdsQ8wyqqyFyknOXjIyD0jM1a?=
 =?us-ascii?Q?HIU2zw9EJzZiIly3CSQXi/vkY9ibjMgE7m/JeQ+mGCupdjj6hpSUYtp5SK5f?=
 =?us-ascii?Q?7EnKusAF2KSgF4YripzlgOmqDC2vXfyHHR0eF3TBG5mL5syGVbc8Tdct/UcA?=
 =?us-ascii?Q?+nVJoIy/VoZX4cxu5h4ea9Fz1bdJj9Xa/zEE3ugJ9IDcuibsMb2V1/EiLW/Z?=
 =?us-ascii?Q?osMXFCsTACbkAs2HC/0CYx0KpQGyLEZ2Qrc6iZeCRceKnVRAU/UO/EHjAfZW?=
 =?us-ascii?Q?nRN3m17e1BI6FxYaKVAh75y18MkmTRDm+/zt/fP6pzKLpNMGU4KAodtZr9d4?=
 =?us-ascii?Q?QympU1JP0siTdZgyhSppF/ifdFQvRjvtntRUG6/qv7g7IEd5L6etG0LbpFON?=
 =?us-ascii?Q?hkHTGBuVvHvd3VBNt4FK8ZvMlFBGVyrP9yTVKKlyVe9iib6mJkDQB8292mFU?=
 =?us-ascii?Q?7wexTyKC3/3vDk1XMdy4I84w4TOw9Y+Juq1x6uVZ+XgamjoJ+wqoOAaGHdP7?=
 =?us-ascii?Q?CczyMo+EoQ/cFvxQtYyZlLyVGgKC3NVkIV7nLfRAj0c/OR3s2zIkxao35uXb?=
 =?us-ascii?Q?zEvztJeALKj13r2R21QxKQrlAaNwuJwrz6lNS6PoiH0rsJmK1frAkvAgcHz7?=
 =?us-ascii?Q?Z1yD2ewacckayLhC6gtxbpD92JDhuGiE+rvVhX5nlSGympNLvs3x95TBokKP?=
 =?us-ascii?Q?v3vU66cqpomJWf0CID/la9+H2K1ddimaqvX7vEGf0ORADbXdh+0zbZGlSXjo?=
 =?us-ascii?Q?Kqo+rp9uY5NB5QgzwLwDqU8ONHtZZxwuvDrHxtv1LFVDCv8pXyXKnnm0u6J9?=
 =?us-ascii?Q?2vPqmfL8gtMuNIoLDwcfZvU3+AHy4Bgdg7Pqg5F9IvHooO1OaOU5z3fh/KI0?=
 =?us-ascii?Q?c+0aZgjeo7gLrFxyNSKo7SWEqCYBvq5m2FbM30usGJnadfPY5umtKLhOWg0X?=
 =?us-ascii?Q?hn8KfMDObf9Z8JmfdyShhpvQ8KrD99/Rxu5jYR7n/z8iGfnKpf5Bhp9bpKFL?=
 =?us-ascii?Q?jA1A8mwpyy3DqPfk7fAFiW1q5HfJ9Em6rANvm/CfDi2AAmiomztNLvUfxvd/?=
 =?us-ascii?Q?3mWFDxSv6kmyuZuzSg4T32xBwNh2MCuo0PHtRKXYdD/aH2aXVg5oZipznFvr?=
 =?us-ascii?Q?zxIMP1tOnzxK3cYsL40XuYyXDzCiLWrr/2srkrwjkyAWEJtFbvJ059QY4zI6?=
 =?us-ascii?Q?/EPXRhYJDBK0ItSUMuxbLQCEzfm7OAIAHklSWaWhTrAqAZhqWMJqSpGrvvBT?=
 =?us-ascii?Q?mToFIbM6nHiMUzU46MdTnrqDRTRFJZR/7M5swLGS2lkShQXOqvyHQGAizOxl?=
 =?us-ascii?Q?yAzktoUGy2s/K2CJJZogWhsW+YBVssvvDZfgTzmZe5kOk3bDs/yDDGkFodGT?=
 =?us-ascii?Q?JPamFQQA3Rln772nJUHFbPpdtF9Vs1Gs+LSmHdy8iSHpWIRNsoLl1tUp4Ycg?=
 =?us-ascii?Q?i+cUh7NzlQvU8yxDb39/elXCTWZhrHs6fs2nlHWuAA4s2Fv6ESlsqLuYgTfd?=
 =?us-ascii?Q?/NcDi/UtwY5Vp26qQVn4KzPFPyIV9EsjPiYddOjPMmMifa2FBqLUnMg4I71X?=
 =?us-ascii?Q?rE6VtzhdizD8IbDKBqZSw4AGyamog5ZJoQOkUPSX563E?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e2055aa6-bbd0-41ed-f61f-08da5a499b64
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6311.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2022 03:35:41.4106
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HL1WkJRCeClD8U2NNoR/a3vUVoBbnXogfXAORIolmjaZ4FGrTxdlO4ptsRddeLXMmuDIx+JA1dN/reIFXJxDMw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1325
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jun 28, 2022 at 03:47:27PM +0100, Jonathan Cameron wrote:
> On Mon, 27 Jun 2022 21:15:24 -0700
> ira.weiny@intel.com wrote:
> 
> > From: Ira Weiny <ira.weiny@intel.com>
> > 
> > The OS will need CDAT data from CXL devices to properly set up
> > interleave sets.  Currently this is supported through a DOE mailbox
> > which supports CDAT.
> > 
> > Search the DOE mailboxes available, query CDAT data, and cache the data
> > for later parsing.
> > 
> > Provide a sysfs binary attribute to allow dumping of the CDAT.
> > 
> > Binary dumping is modeled on /sys/firmware/ACPI/tables/
> > 
> > The ability to dump this table will be very useful for emulation of real
> > devices once they become available as QEMU CXL type 3 device emulation will
> > be able to load this file in.
> > 
> > This does not support table updates at runtime. It will always provide
> > whatever was there when first cached. Handling of table updates can be
> > implemented later.
> > 
> > Finally create a complete list of CDAT defines within cdat.h for code
> > wishing to decode the CDAT table.
> > 
> > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Co-developed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> One query inline, though I'm fine with it either way, just want to
> understand your logic in keeping completion when Dan suggested using
> flush_work() to achieve the same thing.
> 
> > 
> > ---
> > Changes from V11:
> > 	Adjust for the use of DOE mailbox xarray
> > 	Dan Williams:
> > 		Remove unnecessary get/put device
> > 		Use new BIN_ATTR_ADMIN_RO macro
> > 		Flag that CDAT was supported
> > 			If there is a read error then the CDAT sysfs
> > 			will return a 0 length entry
> > 
> ...
> > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > index c4c99ff7b55e..4bd479ec0253 100644
> > --- a/drivers/cxl/core/pci.c
> > +++ b/drivers/cxl/core/pci.c
> > @@ -4,10 +4,12 @@
> 
> > +static int cxl_cdat_get_length(struct device *dev,
> > +			       struct pci_doe_mb *cdat_mb,
> > +			       size_t *length)
> > +{
> > +	u32 cdat_request_pl = CDAT_DOE_REQ(0);
> > +	u32 cdat_response_pl[32];
> > +	DECLARE_COMPLETION_ONSTACK(c);
> > +	struct pci_doe_task task = {
> > +		.prot.vid = PCI_DVSEC_VENDOR_ID_CXL,
> > +		.prot.type = CXL_DOE_PROTOCOL_TABLE_ACCESS,
> > +		.request_pl = &cdat_request_pl,
> > +		.request_pl_sz = sizeof(cdat_request_pl),
> > +		.response_pl = cdat_response_pl,
> > +		.response_pl_sz = sizeof(cdat_response_pl),
> > +		.complete = cxl_doe_task_complete,
> > +		.private = &c,
> > +	};
> > +	int rc = 0;
> > +
> > +	rc = pci_doe_submit_task(cdat_mb, &task);
> > +	if (rc < 0) {
> > +		dev_err(dev, "DOE submit failed: %d", rc);
> > +		return rc;
> > +	}
> > +	wait_for_completion(&c);
> 
> Dan mentioned in his review that we could just use
> flush_work() and drop the completion logic and callback.
> Why didn't you go that way?  Would want to wrap it up
> in pci_doe_wait_task() or something like that.

In early reviews of the Aux Bus work Dan also specified the above design
pattern.

	"I would expect that the caller of this routine [pci_doe_exchange_sync]
	would want to specify the task and end_task() callback and use that as
	the completion signal.  It may also want "no wait" behavior where it is
	prepared for the DOE result to come back sometime later. With that
	change the exchange fields can move into the task directly."

	-- https://lore.kernel.org/linux-cxl/CAPcyv4hYAgyf-WcArGvbWHAJgc5+p=OO_6ah_dXJhNM5cXcVTw@mail.gmail.com/

I've renamed pci_doe_exchange_sync() pci_doe_submit_task() because of other
feedback.  Here the callback is set to cxl_doe_task_complete() and we have to
wait because this particular query needs the length prior to the next task
being issued.

I use flush_workqueue() within the shutdown handling (or if someone destroys
the mailbox or abort fails) to first block new work from being submitted
(PCI_DOE_FLAG_DEAD), cancel the running work if needed (PCI_DOE_FLAG_CANCEL
[was ABORT]), and then flush the queue causing all the pending work to error
when seeing PCI_DOE_FLAG_DEAD.

Ira

> 
> > +
> > +	if (task.rv < 1)
> > +		return -EIO;
> > +
> > +	*length = cdat_response_pl[1];
> > +	dev_dbg(dev, "CDAT length %zu\n", *length);
> > +
> > +	return rc;
> > +}
> > +