Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 16 Aug 2022 11:49:59 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Davidlohr Bueso <dave@stgolabs.net>,
        Dan Williams <dan.j.williams@intel.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        <linux-cxl@vger.kernel.org>, <nvdimm@lists.linux.dev>,
        <bwidawsk@kernel.org>, <ira.weiny@intel.com>,
        <vishal.l.verma@intel.com>, <alison.schofield@intel.com>,
        <a.manzanares@samsung.com>, <linux-arch@vger.kernel.org>,
        Arnd Bergmann <arnd@arndb.de>,
        <linux-arm-kernel@lists.infradead.org>, <bp@alien8.de>,
        <x86@kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] arch/cacheflush: Introduce flush_all_caches()
Message-ID: <62fbe6d7b75ae_f2f5129482@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220803183729.00002183@huawei.com>
 <9f3705e1-de21-0f3c-12af-fd011b6d613d@intel.com>
 <YvO8pP7NUOdH17MM@FVFF77S0Q05N>
 <62f40fba338af_3ce6829466@dwillia2-xfh.jf.intel.com.notmuch>
 <20220815160706.tqd42dv24tgb7x7y@offworld>
 <Yvtc2u1J/qip8za9@worktop.programming.kicks-ass.net>
 <62fbcae511ec1_dfbc129453@dwillia2-xfh.jf.intel.com.notmuch>
 <20220816165301.4m4w6zsse62z4hxz@offworld>
 <CAA9_cmfBubQe6EGk5+wjotvofZavfjFud-JMPW13Au0gpAcWog@mail.gmail.com>
 <20220816175259.o5h5wv23rs2bvcu6@offworld>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220816175259.o5h5wv23rs2bvcu6@offworld>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fmOEV0HyL7kuANbLl5GvlXJ1+gtIRaufKnSpDjpftSCM/3SWfonkrKqdjr1B?=
 =?us-ascii?Q?D91lnb6DZhVcSsUu71BnX0Oa3e234PDlIKcCoK8l5BKohkD8xapvINsQqUiY?=
 =?us-ascii?Q?tLZ3F3pjRYmtddyfO96F1Vp6BZ+9Wo27A/UrxlkOWCCck7AdCuct3o537Bng?=
 =?us-ascii?Q?sgLIZkySVMY5opYP6AYESre3gv5LOJ3vP7zrbf9kcl6gTWjiCEB76nWY41rJ?=
 =?us-ascii?Q?4UEviaK5K4/pEvFnEgyD6uTZRzsbv/wmRTV8lW+Itv5JqP4rgroyffo3piVW?=
 =?us-ascii?Q?JM3BLt1NK6v74lrLBNzviZPvpgWVeVJoqRoDp1OJQXELmRmSxxXXDbQNMebg?=
 =?us-ascii?Q?JYjh0Xrf83fvPDiaWu23S7yEEbI9iWJ+irsfPPjUgjHGNbyBynfk9CYta4SR?=
 =?us-ascii?Q?2JOIB+OEZkJv806IlJdS7Crjp1g5pPwPb9+/YdIiYzYA4xZ5MC+qZENqtmQL?=
 =?us-ascii?Q?9cs2I7il8nROzfxxEh2zd4nLRAjA5tX2R8fRQ4NvTiRPhtaho8cwWDeADRAN?=
 =?us-ascii?Q?8A21Fp5UY7RuNpjXOIRgy96pKFf6vTNOpTPErQylmCa/D6eDgEskTreXZkL3?=
 =?us-ascii?Q?8W1mKr6ssmZd6dfSL6yQQgJ2nZ+oyWd6GfNv+EPg4LuHCYRvVD1ImgCM2YrM?=
 =?us-ascii?Q?765lyKawqINY4g3xh5r0UFIQ2PTx6xD4+ZWus+cNFlljA0c0Visq52bq8U9+?=
 =?us-ascii?Q?y0enUYo14P3LbQpuYVuIg5Fkv6ucnIF4C1r5eDTnP96hY1VuYSN6UK1vkIjP?=
 =?us-ascii?Q?gGkgN8tuVwbtsky6hTDNOmkwcy8m493n28LZdgldwixbseFXAaSNGTxyRD6x?=
 =?us-ascii?Q?UUUnOGR75cc95I/Bwj1CyvUMEFVAPmgdYuBGxr8RPsSzNhFdjfJV1+QmCGHp?=
 =?us-ascii?Q?3gJ0btY4vAt1VP1zafLpohZRy6eCR0aZhsdKzMM/WZrKt+8M3nxmxRgTry05?=
 =?us-ascii?Q?HTRyL5TlENOEc8W9mUnrOSzMW0UEaD2XRmuvQ0bbL3JvhXpVji7QvZG8B8cK?=
 =?us-ascii?Q?WfWDD3X0wLzX+tNN93kuecAeDowNQWMR9RQmG6E3XgtFd1ZaGj5SNPpPYI9G?=
 =?us-ascii?Q?yJdmU0smxKLt+C4KWnQ5ymG8hr6sImVuBXhT+CBbu5JHsTMJ/cKznnoFBCZj?=
 =?us-ascii?Q?x0hsdRGCLi7oCtoglZrXXFhNB7oue99y5s4NbG+IkxS217Ro2AcRMmC+XiYy?=
 =?us-ascii?Q?+V5m3m9rM/U/3W+RwGImuQnJkzxHHDBr/k+s7Z39csTLq0EvMCYRd+J5LeCM?=
 =?us-ascii?Q?uTxmfqMgUWh+WXiaTSuZvHmEbYTs4zlT+Vy8cfVGZWB6GZ4Y6JCjd3kHK6RX?=
 =?us-ascii?Q?WjVif8nETxPMeqnQS/CItX9CfNoLvEd9he548VnhsLbCQXF/4o5Pb5NeUayK?=
 =?us-ascii?Q?gkaF2DPl7f+5uZHRIZrGMI8x9ZsaAxa17sQGow1TB+k0H28kwYz/SOyrKM+7?=
 =?us-ascii?Q?sptznEJ1TWecBVA9xEHQnLx5v/paSSZ2AtM9AcnVTkZgVN3v2Nat7CG9o5ld?=
 =?us-ascii?Q?dOUP9bWKwKGhyufBdCNSy1hB4w+I6/J4SlHm7/hHV2ecFZaCwlI8RKagSrPw?=
 =?us-ascii?Q?/PkBI4xTqn4uPztDmKAsA9bAHhl175c1Lpff6Vtwzfe0hgD5YUnklduQRlPl?=
 =?us-ascii?Q?OQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 727a576d-d259-43fd-808c-08da7fb820b7
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2022 18:50:02.8140
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: A75Km2RItXdiJQYU5RiBfAOggyjxBqGj2dwFMIe4RXaK/TJpRsN05ZkMzj7xJ4s++zzNEREpQPflbXQxiTeUzxHR2hIg2Fq8VDVNrhKX9kk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3228
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Davidlohr Bueso wrote:
> On Tue, 16 Aug 2022, Dan Williams wrote:
> 
> >On Tue, Aug 16, 2022 at 10:30 AM Davidlohr Bueso <dave@stgolabs.net> wrote:
> >>
> >> On Tue, 16 Aug 2022, Dan Williams wrote:
> >>
> >> >Peter Zijlstra wrote:
> >> >> On Mon, Aug 15, 2022 at 09:07:06AM -0700, Davidlohr Bueso wrote:
> >> >> > diff --git a/arch/x86/include/asm/cacheflush.h b/arch/x86/include/asm/cacheflush.h
> >> >> > index b192d917a6d0..ce2ec9556093 100644
> >> >> > --- a/arch/x86/include/asm/cacheflush.h
> >> >> > +++ b/arch/x86/include/asm/cacheflush.h
> >> >> > @@ -10,4 +10,7 @@
> >> >> >
> >> >> >  void clflush_cache_range(void *addr, unsigned int size);
> >> >> >
> >> >> > +#define flush_all_caches() \
> >> >> > +  do { wbinvd_on_all_cpus(); } while(0)
> >> >> > +
> >> >>
> >> >> This is horrific... we've done our utmost best to remove all WBINVD
> >> >> usage and here you're adding it back in the most horrible form possible
> >> >> ?!?
> >> >>
> >> >> Please don't do this, do *NOT* use WBINVD.
> >> >
> >> >Unfortunately there are a few good options here, and the changelog did
> >> >not make clear that this is continuing legacy [1], not adding new wbinvd
> >> >usage.
> >>
> >> While I was hoping that it was obvious from the intel.c changes that this
> >> was not a new wbinvd, I can certainly improve the changelog with the below.
> >
> >I also think this cache_flush_region() API wants a prominent comment
> >clarifying the limited applicability of this API. I.e. that it is not
> >for general purpose usage, not for VMs, and only for select bare metal
> >scenarios that instantaneously invalidate wide swaths of memory.
> >Otherwise, I can now see how this looks like a potentially scary
> >expansion of the usage of wbinvd.
> 
> Sure.
> 
> Also, in the future we might be able to bypass this hammer in the presence
> of persistent cpu caches.

What would have helped is if the secure-erase and unlock definition in
the specification mandated that the device emit cache invalidations for
everything it has mapped when it is erased. However, that has some
holes, and it also makes me think there is a gap in the current region
provisioning code. If I have device-A mapped at physical-address-X and then
tear that down and instantiate device-B at that same physical address
there needs to be CPU cache invalidation between those 2 events.