Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 23 Nov 2022 09:39:38 -0800
From:   Ira Weiny <ira.weiny@intel.com>
To:     Lukas Wunner <lukas@wunner.de>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Alison Schofield <alison.schofield@intel.com>,
        "Vishal Verma" <vishal.l.verma@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Gregory Price <gregory.price@memverge.com>,
        "Li, Ming" <ming4.li@intel.com>, <linux-pci@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH V2 1/2] PCI/DOE: Remove the pci_doe_flush_mb() call
Message-ID: <Y35a2nhgOUPhhLeS@iweiny-desk3>
References: <20221122155324.1878416-1-ira.weiny@intel.com>
 <20221122155324.1878416-2-ira.weiny@intel.com>
 <20221122195316.GC20515@wunner.de>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221122195316.GC20515@wunner.de>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tPRSX1D1yOen+0vAfgDsTm2yasZZQiHZpmBnGpJFt0OPbqKsoOm5ie19hsbp?=
 =?us-ascii?Q?C5TeJxGHfSPuOOu47HxFKoGT9wKm8Y/uVZd6AabpyO9AerpifEtLl1wOuUXm?=
 =?us-ascii?Q?xYPyEsAkcv1qWyea94J5lCoQ9SOsADXle9HvkBtQ8LEzlL10YbW7vvHM3rKO?=
 =?us-ascii?Q?z6vtIEQ/uTP+ifZvgdCJtLQkBXaYli8a9d78cDfSckto412W1+SETnOgBDY+?=
 =?us-ascii?Q?wFWzR40T7BJU0m7b42x+sDlk9ji6MLdjv7pYHye/aAvKlsWGCBaDkRVjhgOJ?=
 =?us-ascii?Q?Weft83lQiSB6VS01AA+JMnMpmVkWmW4v1GmCNHZi4hofwzqeLkqvRSt3uoK5?=
 =?us-ascii?Q?MRfOqc7RUmza2RDTf6Gaj5BCNpLEmBBhiogI8Stxynz/FO5RZTqYYwGaE6e5?=
 =?us-ascii?Q?kDOZexYkI85eZ6W1ow6r8kMrVa8ZaxqhYtHFn3AscGJCVtvGaK8HQgHIsG4t?=
 =?us-ascii?Q?ukJzMEy3pD8NJ1vpHVLc4GEfFk8/4nnJBcNRi5X49QXCxR5chz5+5TBrZaiP?=
 =?us-ascii?Q?PYWA0Vcb+fPCNQKvOzvtNmnLjW7L7RR5SDHtIFqmQR70+8mHQpW3IW6JDMk3?=
 =?us-ascii?Q?Zf+dfJxX3PtxqGFerRIZvkZ+AAK4lsN51sE0O0a+CGy15nVy46Eyz2x2bWGh?=
 =?us-ascii?Q?5i7QEZu1MEQ0FB0tmJWzT7bNAfTmeI0j312KaCMAqoE3gKWQguroVt9MD/fi?=
 =?us-ascii?Q?9gtUCYkmAbNNkxZ9o9YUnDy9yQhYNn8fIObFoOVD/75PlFsHR3HNGUVPBLWz?=
 =?us-ascii?Q?UXvI6mUK+zH0LpXxbNJWoOxiq3w3RSjCDCQyCtRKHMSMX+HQ7UwbKeSaB15g?=
 =?us-ascii?Q?Rr8aksPDQAR4CNKYOITT/MvPPyNyR276ZHRL4Ic7994Wldr3D4iExzN0PdDI?=
 =?us-ascii?Q?F9WbHkOrzbKNnN1u/iyNTRJD1XBeWf3qku3a/ScX5CN5oSAjgVx+qOMyidyq?=
 =?us-ascii?Q?mewmL+Yq7D9mUUgith2tTjKlyzvzRt7p0Rk92jRIWfTHj3ZlWvHPk3HXtjF9?=
 =?us-ascii?Q?aorBQWrjgKrX2QGWPaJXC9kaPGRF4cH+mawCcLHP7p1rvzsVZaJltaPggj9y?=
 =?us-ascii?Q?Vk7PLWnxB4yurtgGVPgqUlO9c/C/BwzrVhs5B8yL1XcgG4jZbXf0vsHxj3Te?=
 =?us-ascii?Q?S69/BMEDP+GQ3eZftKNShwjoyZUvn8YSdnTdqcZkpx5cYFCPxnRLCPFlf/ea?=
 =?us-ascii?Q?6SuhW1ii6oK3DvA3tDnDyDuk38P1mSFoX3Gbk3rtxDFNlqYvgzgN3jJwb3pI?=
 =?us-ascii?Q?P2d5ak3yw+qSXRHjubtEKTeJrGTDccr4k96oVuVIW0YyiEFe8aNS0hZ7YRWt?=
 =?us-ascii?Q?dRAl3TqH7+JV6tVbmR+9QA6GXmo97pOgwe/C/rpDVzDuEki7BCAZFkSAQAow?=
 =?us-ascii?Q?JB/lhYAvmoD/f+3CZ/mLBATAGOIaOp1gG465PBer7j0cG1QySYvOoe55fN+C?=
 =?us-ascii?Q?Ls9xwum7ry677Si1KsetzMMS7M5Fjvznh/rAv1Hns9XosO21Fj4FlhnAKb/W?=
 =?us-ascii?Q?ALLdQDWNpwLhyIGSuCoINv0n5h7krYjZq/u0xUSjCV5Z5fCAg9ysPKzjmJSd?=
 =?us-ascii?Q?ZdSTQpg6avch0Gco4JrkMXOlP75WbNAR6MqJHyR+?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c74209f-deac-4042-e4c0-08dacd79b475
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2022 17:39:43.0895
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: japuLfvNhCreLq5V8FXQ0xY9u97oeuOgZtMuBo93A34GcXLXDw2TKOo/V3BLorxT1RFI4tNRV9HqJKcQ2OqSBg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR11MB5591
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 22, 2022 at 08:53:16PM +0100, Lukas Wunner wrote:
> On Tue, Nov 22, 2022 at 07:53:23AM -0800, ira.weiny@intel.com wrote:
> > Each struct doe_mb is managed as part of the PCI device.  They can't go
> > away as long as the PCI device exists.  pci_doe_flush_mb() was set up to
> > flush the workqueue and prevent any further submissions to the mailboxes
> > when the PCI device goes away.  Unfortunately, this was fundamentally
> > flawed.  There was no guarantee that a struct doe_mb remained after
> > pci_doe_flush_mb() returned.  Therefore, the doe_mb state could be
> > invalid when those threads waiting on the workqueue were flushed.
> > 
> > Fortunately the current code is safe because all callers make a
> > synchronous call to pci_doe_submit_task() and maintain a reference on the
> > PCI device.
> > 
> > For these reasons, pci_doe_flush_mb() will never be called while tasks
> > are being processed and there is no use for it.
> 
> Going forward my plan is to allocate all existing DOE mailboxes
> of a device upon enumeration.  That will allow concurrent use
> of a mailbox by multiple drivers.
> 
> When a pci_dev goes away, say, because it's been hot-removed,
> we need a way to abort all ongoing DOE exchanges.
> 
> pci_doe_flush_mb() seems to do just that so I'm not sure why
> it's being removed?

I guess I was not explicit enough in the commit message.  1) it is not used
today.  More importantly 2) it does not work.

Because we are not using it currently I thought it best to remove it rather
than try and fix it.  Once we have a use then we can figure out how to make
sure the doe_mb[*] is valid until all tasks are flushed.

Ira

[*] Probably with a reference on the struct as was discussed before.

	-- https://lore.kernel.org/all/20221122094627.00003f2c@Huawei.com/

> 
> Thanks,
> 
> Lukas